Mirror Mirror

Inspiration

We wanted to make a laptop screen feel like a window into a world you could build with someone else. Games and creative tools let us explore incredible 3D spaces, but interacting with them with a keyboard falls short of full immersion and buying specialized hardware is very expensive and difficult to learn.

Mirror Mirror started with a question: what if you could look at something to select it, reach out to move it, and describe something to bring it into existence? We set out to explore that using a pair of eye-gaze tracking glasses, webcam, and a shared virtual world.

What it does

Mirror Mirror is a collaborative 3D sandbox controlled through your eyes, hands, and voice.

Choose a set, from a peaceful Japanese garden to a neon cyberpunk city, and explore it as a cartoon avatar either alone or with others. Using hand gestures to control movement and look around. In edit mode, gaze targets objects, pinching grabs and moves them, and spreading two pinched hands resizes them.

You can also describe an object aloud, “the Hogwarts castle,” for example, and generate a version inside the scene, powered by Meta Muse API. Generated objects can be saved to a local library and reused.

The shared workspace supports multiple users over a local network, synchronizing objects, avatars, and tracked hands. Head tracking adds a depth effect that shifts the scene’s perspective as you move.

How we built it

We split Mirror Mirror into four connected pieces:

Tracking: Python, OpenCV, and MediaPipe capture webcam and glasses input and estimate hand landmarks, iris features, and head pose. Filtering smooths the signals, while gaze calibration maps eye features to screen coordinates.
The world: Three.js renders the environments, avatars, and interactive objects. Browser-side gesture state machines translate hand poses into movement and editing actions.
Voice creation: Muse Voice Transcribe converts speech to text, and Muse Wonder turns the description into a structured assembly of boxes, spheres, cylinders, and cones. Python validates and positions the parts before Three.js renders them.
Collaboration: A Python WebSocket relay connects the browsers. Object ownership and host-authoritative scene synchronization coordinate edits and reconstruct the scene for users who join later.

Tracking and voice generation run independently so waiting for an AI response does not interrupt hand input.

Challenges we ran into

The biggest challenge we faced was in the department for making the webcam and glasses input sync together. There were lots of issues with the glasses camera ignoring the glare and properly calibrating to the viewport of the laptop screen. From our cameras breaking at first to not having the right adapter, we had to think on our feet and find a quick solution to our problems on the fly. It took a lot of fine tuning and endlessly testing our software and hardware integration. At the end of the day with careful patience we were able to work together as a team to solve our issues. We calibrated our cameras with opencv to mask out the glare and integrate the head tracking with the positioning of the eyes to create an intuitive feel to the experience.

Gestures also needed clear boundaries. A hand movement meant for navigation should not grab an object, so we introduced separate MOVE and EDIT modes with explicit switching and visual/audio feedback.

Voice generation brought a different problem. An object description does not automatically produce sensible geometry. We added schema validation, spatial relationships, and a repair attempt to help keep generated assemblies connected and correctly proportioned.

Collaboration required handling simultaneous grabs, delayed messages, and reloads without letting the two scenes drift apart through addressing race conditions.

Accomplishments that we're proud of

We’re proud of connecting sensing, generation, rendering, hardware inputs, and collaboration into one creative experience. A generated object becomes part of the workspace immediately. It can be targeted, grabbed, resized, and shared.

We also built beyond the core interaction, adding themed environments, animated avatars and objects, an object library, and head-responsive perspective. Automated tests and two-browser checks helped us verify synchronization and interaction logic as those pieces came together.

What we learned

We learned that natural interaction takes deliberate design. Recognizing a gesture is only the beginning. Timing, feedback, and recovery determine whether it feels usable.

We also learned that constrained AI output is powerful. Giving the model a small vocabulary of shapes and clear spatial rules made its output practical to render, edit, and synchronize.

Separating sensing from interaction logic helped us tune controls quickly while keeping the rest of the application stable.

What's next for Mirror Mirror

Our next priority is testing with more people and measuring gaze accuracy across different lighting, camera positions, and users. We want to improve calibration, fast-motion recovery, and hand tracking at difficult angles.

From there, we’d like to add persistent shared worlds, collaboration beyond the local network, richer object editing, and easier environment switching.