Short Summary

Main Idea: Decorate your space with beautiful artworks and have your own personalized and evolving art gallery.

Outlook / Vision: App should be a spatial application, always open when walking around room with AR glasses used casually as decoration and check out new artworks every day.

What we implemented: Condensed version of this idea for the Quest 3: Place and move frames, rate paintings with thumbs up/down, get new paintings recommended to your preferences. Paintings change quickly to show how the app adapts to your preferences. Frames stay between sessions through spatial anchors.

Full Write-up:

Inspiration

The inspiration for "ArtVerse" came from a desire to blend the physical and digital worlds, allowing users to interact with art in a completely new way. With the rise of Mixed Reality (MR) technology, we saw an opportunity to create a platform where individuals could personalize their living spaces with virtual art pieces and discover new art tailored to their tastes. Future AR glasses that enable these virtual paintings to exist in your space at all times inspired us to realize this concept on the Meta Quest 3.

What it does

Artverse allows you to decorate your space with beautiful artworks and be able to have your own personalized and evolving art gallery, showing you new paintings and adapting to your tastes.

When users launch the ArtVerse MR application, they can start decorating their surroundings by pinching and placing virtual wall frames onto real walls, adjusting the size as needed. Once the frames are in place, they get filled with various artworks of different eras, styles, etc. Users can interact with the art by giving it a thumbs up or down, and the application recognizes these interactions, saving the user's preferences. Through this feedback, the app can learn the user's tastes and suggest artwork that aligns with their preferences in the future. Walking closer to an artwork also reveals a panel with information about the piece: Its title, who made it, a short biography of the artist and the year it was made.

All wall frames get saved as spatial anchors and return the next time the user starts the app.

How We built it

ArtVerse was developed entirely using Unity, with 3D models created in Blender and the prototype initially designed in ShapeXR. These elements were then integrated into Unity to bring the project to life.

We made use of passthrough, scene understanding, spatial anchors, and the interaction SDK of the meta presence platform to fulfill the Mixed Reality needs of our application.

Challenges We ran into

Hand-based Interactions: We wanted the entire experience to be usable without the use of controllers and with minimal UI getting in the way. This left us with quite a few tricky choices as to how we let users manipulate and interact with the app naturally without using any controller buttons or any overbearing UI.
Creation of variable size frames, correct scale of paintings inside the frame, and proper placement of the UI around the paintings.

Accomplishments that we are proud of

Creating a functioning app with most of the features we initially hoped for, including:
Successfully using Spatial Anchors to save the frame positions across sessions.
Having natural interactions for creating frames and giving feedback to artworks.

What We learned

Building "ArtVerse" was an incredible learning experience. Some of the key things We learned include:

User Experience Design: Designing for MR requires a different approach to user experience, focusing on intuitive interactions and spatial awareness.
How to better manage several people working on the same project, juggling tasks and making sure there is little overlap.

What's next for ArtVerse

We have several features we started, but could not quite get into the final version. This includes having sculptures and not just artworks, which can be displayed on tables (real and virtual) or inside wall frames that act as a window into a small virtual space containing the sculpture. We have an experimental scene containing wall frames that open windows into 3d videos, that we weren't quite able to integrate into the main experience. We also experimented with generating art through generative AI, both 2d paintings and 3d models, but ultimately decided to steer away from AI generated art and instead focus solely on traditional artworks.

We also wanted to implement social features, more specifically a function to comment on the artworks and share your perception of what makes an artwork great, or maybe not so great for you.