Mirror-Mirror

Inspiration

This game was inspired by the meshing of two fairy tales: "mirror, mirror, on the wall" from Snow White and "the Beanstalk" from Jack and the Beanstalk. We decided to make use of computer vision to make a fun game where two people try to sync up their faces and facial expressions as much as possible so that Jack can climb the beanstalk to success.

What it does

Two players sit side by side (or are on separate computers) and copy each other's faces in real time. The better they mirror each other, the higher Jack climbs a beanstalk. Along the way:

• Live feedback: it shows how in sync you are, who is leading and who is following, and how big the delay is.
• Prompts: timed prompt cards keep you moving, such as "Big smiles", "Left player leads" and "Swap!".
• Recap: after the round, a screen shows time in sync, your longest streak, who led, and your best-matched expressions. Every number comes straight from the measurements.
• Leaderboard: a live leaderboard display shows the top climbs.
• Remote play: you can play on two computers.
• Everything runs in the browser, and no video leaves your computer.

How we built it

• Tech: TypeScript and Vite, with the beanstalk scene drawn on a browser canvas.
• Face tracking: Google's MediaPipe tracks two faces in the browser. We turned its 52 face measurements into 18 expression channels plus head movement.
• Sync measurement: we compare the two players' movements over a 4-second window, allowing up to a 1-second delay, and use the best match as the sync score. The delay's direction says who led. Two still faces score 0, and different expressions score lower, so a high score needs real mirroring.
• Leaderboard and remote play: a small Node server stores scores and updates the display live. Remote play sends only the face numbers between computers, never the video. (This means that the players will see a drawing representing the other player's facial expression, not the actual face itself, to save data, maintain privacy, and help players know what to focus on).
• Tuning tools: every round can be recorded and replayed, and a tuner page re-scores recordings while you move the settings.
• Tests: 119 automated tests ran by Claude.

Challenges we ran into

• Making the score honest. Early ideas rewarded two still faces, or different faces that happened to overlap by chance. We built the score so stillness earns nothing and mismatched expressions lower it.
• Left versus right. When you mirror someone, your left brow answers their right brow. We fold left and right together so mirroring and copying score the same.
• Real cameras. A laptop can list a phone's virtual camera first. The game pointed at it and hung, so we added camera selection and plain error messages.
• Timing noise. Live play and replay of the same round differ by up to about 0.3 m, from timing jitter. We measured it instead of assuming a cause.
• Two computers. Browsers only allow cameras on secure pages, and campus Wi-Fi can block devices from reaching each other.

Accomplishments that we're proud of

• A sync measure that rewards real mirroring and is covered by tests.
• A full game loop from lobby to recap, with a leaderboard and a live display.
• Remote play across two browsers: in our simulated test, the host and guest ended with identical recaps.
• A privacy-first design, since video never leaves the browser.
• A tuning tool built so settings can be adjusted from real recordings.

What we learned

• Measure before trusting a theory: one fix we expected to explain a gap explained about a twentieth of it.
• A fair score depends as much on what you don't count as on what you do.
• Browser cameras, permissions and virtual devices cause surprising problems, so the page should always say what it's waiting for. (This took us far more time than we thought it would)

What's next for Mirror-Mirror

• Fine-tune the sync settings on real faces. They were originally set using simulated data, and sometimes two matching faces don't sync.
• Move the leaderboard to a hosted database and a public server.
• Add hand-made art and sound, and more round styles.
• Keep playtesting remote play on real networks.

Built With

• javascript
• typescript