Inspiration
Kids ask why all day. Why does a dam need a thick bottom? Why does a kite go up? Books and videos answer with words, but these are things you understand by doing them: you feel the water push, you tilt the kite and watch it climb. Our bar was a paper plane: fold it, throw it, and see why it flies farther.
Mixed reality can put that kind of experiment in any room, and AI agents can now write real interactive software. So we asked: what if every question could become its own small game you play with your hands, made for you while you wait?
It turned out that "AI writes a game" and "AI writes a game you play with your hands, in your room" are very different problems. Most of our work went into the gap between the two.
What it does
- Ask anything. Type a question on the start page, or pinch Make a game in the headset and say it. A card appears at once and shows the game being made.
- Muse's team builds it. A designer picks one idea, one action and a goal, and draws two planning pictures. A coding agent writes the game, including its hand interactions, and plays it in the real headset app with emulated hands to test it, then fixes what it finds. A grader checks it against the request. A gate then runs it locked down, plays it through with a simulated hand, and only then lets a headset load it.
- Play it in your room. Desk games sit on your real table in passthrough. World games put you in a place you walk around: a forest river, a windy hilltop, a desert canyon. Everything is hands-first: pinch, poke, grab, carry and throw.
- Learn by doing. Each game teaches one real idea through its main action, with visible feedback: pressure arrows that grow with depth, airflow ribbons and a lift arrow, ink trails that show a straight flight or a curl.
- Talk to Muse. Muse is also a voice tutor. It answers questions about the game that is open, quizzes you out loud when you ask, and can plan a course of levels.
- A dashboard over your room. In the headset the home screen floats in front of you with no background. Opening a game folds it into a small bar with Home, and Home brings it back. Courses open as folders of levels.
How we built it
- Headset app. WebXR on Meta's Immersive Web SDK (IWSDK) with three.js and uikit panels, on Quest 3. Its host mounts each generated game as an ES module and gives it a kit: hands (fingertip, palm, pinch), grabbable and pressable objects on IWSDK's grab system, panels and sound. It then places the game on your desk or as a world around you.
- Server. Python with FastAPI. Each flow is a LangGraph graph with checkpoints. The coder is a Deep Agent in a Modal sandbox, one per run, with the network blocked and no secrets. It works from skills we wrote for it: learning games, hand play, nine visual looks, game patterns, motion, sound and panels.
- The agent writes the hand interactions, and proves they work. No person codes how the hands work in any game. Every game declares, as data, the gesture that plays each of its moves. An automatic play-through then plays the game with a simulated hand: at 72 and 30 frames per second, again after Try again, and once with no hand at all.
- A real headset in the sandbox. The sandbox runs the real headset app headless through IWSDK's runtime and emulated Quest. The agent opens its game, drives emulated hands from the IWSDK command line, takes screenshots and fixes what looks wrong.
- A gate before every headset. It does static validation, a dry run with the headset's own three.js and kit, a layout check, and the play-through.
- The Studio. It keeps a record of every run: the design, the planning pictures, every agent step, the play-test screenshots and the grader's verdicts.
- Muse, our mascot, is built in code as a small 3D toy with flat colours and an ink outline. It stands beside its question, watches you, waves when pinched, thinks while a game is being made and hops when one is ready.
Challenges we ran into
1. An XR game is not a three.js web app. Language models write good three.js, because the web is full of it, but almost none of that code was written for a headset. A web game has a mouse, a screen and a camera the developer controls. Ours has none of those: the player is a real person half a metre from a real desk, using their real hands, and the scale is real. Our first generated games looked fine in a browser and failed in the headset. Buttons sat behind other objects. Pieces were out of reach. A giant model filled the player's face, and a set piece stood exactly where the player started. The agent had no sense of space; it was guessing coordinates.
How we solved it, in three stages:
- A layout checker. It measures every game from where the player sits or stands: is each hand target visible, in reach and not too big up close? Our first two versions made games worse, cutting scores on 2 of 3 requests in our study. Version 3 removed every blocking layout without hurting looks.
- Research. We studied HOLODECK 2.0, a research system that lays out 3D scenes by having a language model write spatial relations and a solver place the objects. We read the paper and ran their solver ourselves. It works for rooms, but its numbers are room-scale: "near" put a whisk 2.1 m from its mixing bowl.
- A placement solver of our own. The agent now writes a scene plan: each thing's role, its size, and how it stands to the others ("Start near the plane", "the goal far from the player"). A depth-first solver finds places in desk or world units, and it tests every candidate place with the layout checker itself. When a plan can't be placed, it says which thing, which relation and which rule is in the way.
2. An agent that can't see its game writes broken games. Reading code doesn't show you what a player sees. We put the real headset app inside the agent's sandbox, so it plays its own game with emulated hands before finishing.
3. Games that looked right but couldn't be played. A screenshot can't tell you whether a move registers. So the gate now plays every game through, using the game's own gestures. It caught real bugs: in our river dam game, Try again kept the wall from the last round, so the next round was won with no hand at all.
4. Physics written from memory goes wrong the same ways every time. A thrown thing stopped where the hand let go, results changed with the frame rate, and bounces never settled. We built a motion kit: 21 small helpers for throws, bounces, gliding, wind, pendulums, springs, ropes, levers, toppling and floating. Each is tested against its textbook formula and gives the same result at 30 and 72 frames per second.
5. Trusting generated code on someone's face. A game written a minute ago runs in a headset. Every module runs locked down in the gate first: no network, no imports, no DOM, no timers.
6. Memory in the headset. After many games the Quest lagged: about 13 MB stayed alive after each one, held by the panels' layout callbacks and the ECS registries. We measured it on the device, and now release both when a game closes.
7. Scrolling with hands. Dragging a list stopped after a few pixels: the headset input reuses one event object every frame, and the panel library marks it as already scrolled. We clear the mark each frame, and the same drag now scrolls 230 pixels instead of 13.
8. Designing for kids in a headset. Our first look was too faded and our second too blue. The one that worked has no background in the headset, puts every word on its own solid shape, and uses big, bold type.
Research behind it
We didn't guess what makes these games good; we measured it.
- A factor study. 35 games from 11 setups, all from the same four requests, judged blind from the same opening view, plus a layout check and a hands probe on every game. The biggest factor was the model: the same pipeline with a weaker model scored 3.75 lower out of 20. Writing game logic as entities, components and systems improved both games where we tested it.
- Other AI game builders. We studied open-source agents that build web games, to learn what makes them reliable: design documents written before code, skills, and play-testing. We then rebuilt each of those ideas for XR, where the hands, the room and the scale change everything.
- Research systems. We read research layout systems (HOLODECK 2.0) and educational 3D generators, kept what worked, and wrote our own versions for desk and world scale.
Accomplishments that we're proud of
- A sentence becomes a playable, hands-first mixed-reality game in about 15 minutes, with no person in the loop.
- To our knowledge, Muse's Lab is one of the first systems where an AI agent designs, codes and tests the hand interactions of a VR game by itself, and proves with a simulated hand that the game can be played before anyone puts on a headset.
- The four ready-made games were made this way, from one sentence each, and teach real ideas: water pressure, lift, erosion, and how symmetry keeps a plane flying straight.
- Every build is open to inspect, step by step, in the Studio.
- The quality checks are tested themselves: the play-through check has 23 test games that each break one rule on purpose, and the placement solver has 42 checks, including the four example games' scenes.
What we learned
- The model is the biggest factor in game quality, then the pipeline around it.
- Checks beat instructions: a gate, a play-through and a play-test catch what no prompt rule prevents.
- Spatial sense has to be given to the agent. Left alone it guesses coordinates; given roles, sizes and relations plus a solver, it lays out scenes that pass on the first try.
- Testing in the real runtime matters. The agent's own play-tests and our emulator tests found problems that reading the code never would.
- Not every good idea helps. Our first layout checkers made games worse, and we only knew because we measured.
- For young players in a headset, readable beats clever: bold colours, big text, and every word on a solid shape.
What's next for Muse's Lab
- Hand tools and more hand actions as tested kit pieces: pour by tipping, stir, fold, twist, cut, hammer. That opens up cooking, building and making.
- More physics for real subjects: filling and pouring, heat over time for baking, and mechanisms like gears and engines.
- Levels inside each game, so a kid can master an idea step by step.
- Faster builds, so a first playable version arrives in a few minutes and improves while you play.
- Sharing: send a game you made to a friend or a classroom.
- Playing together in the same room.
Built With
- deep-agents
- fastapi
- iwsdk
- langgraph
- modal
- python
- three.js
- typescript
- uikit
- vite
- webxr