Inspiration
Art Museum XR is an existing XR museum app for viewing paintings. For this update, we wanted to bring art appreciation into the comfort of users’ own rooms.
We imagined hanging a favorite painting on the wall, placing a sculpture on a nearby shelf, and bringing an interesting piece closer while relaxing on the sofa.
This inspired us to add a mixed reality mode to Art Museum XR: a way to enjoy art at your own pace, within your everyday surroundings.
What it does
The new MR mode turns your real room into an interactive art museum. You can place paintings around your room and enjoy them while seated. The mode also includes 60 3D artworks available exclusively in MR.
Using gaze and hand interactions, you can select distant artworks, bring them closer, resize them, and browse the artwork selection UI. High-resolution paintings invite close inspection of fine details. Opening your palm summons an AI guide orb. Through real-time conversation, you can ask questions about paintings while viewing them.
How we built it
We built this update in Unity using Meta Interaction SDK, Meta Mixed Reality Utility Kit (MRUK), and real-time conversational AI:
- Gaze- and hand-based interactions for artwork selection, movement, resizing, and UI navigation
- Scene-aware placement of paintings, 3D artworks, and video players using Meta MRUK
- Optimized high-resolution painting rendering within Quest's performance and memory constraints
- Open-palm gesture recognition to summon the AI guide orb
- Real-time conversational AI for natural conversations about artworks
Challenges we ran into
- Making paintings and 3D artworks feel naturally present in real rooms.
- Tuning lighting so 3D artworks feel physically present across different room lighting conditions.
- Bringing selection, placement, resizing, and conversation initiation together through gaze and hand controls.
- Balancing visual detail with Quest’s rendering and memory limits.
- Making distant artworks and UI comfortable to operate while seated.
Accomplishments that we're proud of
- Enabling art appreciation in real rooms in addition to the existing XR museum experience.
- Making distant artwork selection and movement feel natural through combined gaze and hand interactions.
- Pursuing optimizations for high-resolution paintings within Quest’s constraints.
- Letting users create a space for relaxed, seated art appreciation in their own rooms.
What we learned
- How lighting and shading affect the natural presence of 3D artworks in mixed reality
- The importance of comfortable interactions when viewing artworks while seated
- How combining gaze targeting with hand controls makes distant artworks easier to interact with
- How to balance high-resolution image quality with rendering performance and memory constraints on standalone VR hardware
What's next for Art Museum XR
- Expanding the collection of artworks.
- Exploring how conversations with an AI guide could enrich art appreciation.
- Refining intuitive art experiences through gaze, hands, voice, and other forms of interaction.
Log in or sign up for Devpost to join the conversation.