ImagineVR

What it does

ImagineVR is a voice-driven sandbox for Meta Quest 3S. You start in a dark room facing a terminal, describe what you want, and watch the world take shape around you as the instructions appear on screen. You can ask for a landscape, a building, creatures, objects, or something to do. Then you can walk around, grab and throw things, ride them, and keep changing the scene with your voice.

The goal is for each creation to be playable. If you ask for a city, you should be able to travel through it. If you ask for a shooting gallery, the targets should react when hit. You are building a place to spend time in, not just generating a picture of one.

How we built it

We built the VR app in Unity and a FastAPI server to handle each voice request. The server transcribes audio with faster-whisper, sends the request and the current world state to an AI model, then streams back instructions in a format we created called ImagineScript. Unity reads those instructions one at a time, so the world starts changing while the response is still coming in.

ImagineScript gives the model a set of building blocks for terrain, weather, objects, movement, and reactions. When an object is missing from the library, the model can assemble one from simpler shapes. We also built a desktop mode so we could develop and test ideas without putting on the headset for every change.

Challenges we ran into

The toughest problem was discovering that scenery visible in the Unity editor could disappear in a headset build. The build stripped shader variants used by our original rendering approach, even though the logs said the objects had been placed. We changed how scenery is drawn and added checks that inspect what a built player actually renders.

AI output also needed to survive mistakes. Early on, one invalid field could reject an entire creation and leave the user watching an error scroll by. We changed the system to repair or skip bad parts so the rest of the idea can still appear. Keeping the model aware of the current world without sending an enormous prompt every turn was another challenge.

Accomplishments that we're proud of

We’re proud that ImagineVR is a sandbox rather than a collection of prepared scenes. The same building blocks can make a place to explore, a vehicle to ride, or objects that react to what you do. Seeing a spoken idea turn into something you can reach toward in VR is the moment we were working for.

We’re also proud of the less visible work that makes that moment possible: streaming each step into the world, recovering from imperfect instructions, and testing what the headset really draws. Those changes made the experience feel much closer to the open-ended play we imagined.