Inspiration

Sometimes, all we need is someone to talk to, share a random thought with, or simply have around when we're bored. But we don't always have that companion available. Our rooms are filled with objects that have been part of our everyday lives for years. We wanted to explore the idea of bringing a little personality and life into these familiar spaces. That thought led to ROOMMATES – a mixed-reality experience where AI-powered virtual companions become part of your physical environment. Using passthrough MR, users can see their actual room while placing, talking to and playing with virtual characters.

What is ROOMMATES?

ROOMMATES transforms your physical room into a playful space for companionship. Users can scan their room, place virtual characters on real-world furniture, talk to them, play games and interact through hand gestures.

Meet our two companions: Mr. Buddy: A witty, friendly companion who loves conversations and Rock Paper Scissors. Penny: A cheerful companion who enjoys chatting and playing Catch the Ball. They aren't just characters to watch; they're companions you can interact with in your own space.

How we built it

ROOMMATES was developed in Unity 6.3 LTS for Meta Quest 3, using mixed-reality capabilities from the Meta XR SDK. We used the Mixed Reality Utility Kit (MRUK) to understand the user's physical environment and support room scanning, furniture detection and character placement. Passthrough allows users to see and remain connected to their real surroundings while interacting with virtual characters. For conversational AI, we integrated the Gemini API, which powers the characters' responses and distinct personalities. Wit.ai was used for voice recognition, allowing users to communicate naturally through speech. We also developed hand-based interactions and integrated character animations, game mechanics, physics and visual effects to make the experience more engaging. Our focus throughout development was to bring together AI, mixed reality and interaction design in a way that feels natural, comfortable and playful.

Challenges we ran into

1.Understanding Meta's XR ecosystem, SDKs, room understanding and how to build an experience that works with the physical environment. 2.Ensuring passthrough transitions felt natural and comfortable rather than disrupting the experience. 3.Making character animations match the timing of user actions during games, so interactions felt believable and responsive. 4.Ensuring Buddy and Penny maintained their own personalities and voices across conversations and game interactions. 5.Building interactive games with natural character animations, ball movement, collision behavior and believable reactions. 6.Positioning heart animations and confetti accurately in the mixed-reality environment so they appeared in the intended locations. 7.Time management was another major challenge. Balancing design, learning, implementation, testing and debugging as a team of three designers required continuous prioritization.

Accomplishments that we're proud of

We are proud that, despite being a team of three designers without a dedicated developer and having no prior experience building a mixed-reality application, we brought ROOMMATES to life within the competition timeline.

What we learned

Building ROOMMATES was a learning experience from the very beginning. We explored Meta's XR ecosystem and worked with mixed reality for the first time, learning how to move from understanding the technology to designing, developing and testing a complete experience.

Some of our key learnings were: 1.How small visual details can significantly affect immersion in mixed reality. 2.The importance of comfort, clarity and intuitive interactions in MR experiences. 3.How to balance performance and visual fidelity on standalone VR hardware. 4.How users perceive and interact with mixed spaces differently from fully virtual environments.

What's next for Roommates

We want to bring more characters and interactions into ROOMMATES, improve room understanding, enable persistent character placement and allow users to customize personalities. We also envision characters interacting with each other, making the room feel even more alive.

Our goal is simple: to make every day spaces feel less empty and the world a little less lonely.

Built With

Share this project:

Updates

Submission history