SAO

Inspiration

Sword Art Online promised a world you could simply step into. Twenty years of VR later, we still walk through environments that were frozen at build time, talk to NPCs reading fixed scripts, and call it immersion. The recent wave of generative models changed our question from "how do we build a world faster" to "why build it in advance at all". SAO is our answer: a VR experience where the world is generated, decorated, styled and dramatized around the player, live, by voice.

What it does

You put on the headset and speak. A photorealistic Gaussian-splat world (World Labs Marble) is generated and loaded around you. You point your controller at the floor and say "an old wooden barrel here" — an LLM agent interprets the request, calls text-to-3D generation, and the prop lands exactly where you pointed, grabbable in your hands. The same agent can read the screenplay and stage the whole scene by itself: placing characters, choosing key props. Then you say a visual style — Van Gogh, cyberpunk, watercolor — and a generative video model (Decart Reactor X2) restyles every frame you see in real time. Inside this world an AI director runs an interactive mystery drama, "The Pendulum Sanatorium": the character Vera carries a hidden identity and a trust state; she reacts to what you say, how you say it, and what you physically do. Hand her the pocket watch and the story pivots — the world senses prop-to-character contact and feeds it to the narrative engine. A staged experience flow (start menu, world selection, scene setup, style choice, screenplay editing, live play) runs on interactive floating panels you can click with the controller ray, grab, drag and resize.

How we built it

Unity 6 with URP and OpenXR on PICO. Five custom modules behind assembly definitions: an experience-flow state machine with procedurally built world-space UI panels; an AI Director module where a DeepSeek-driven director LLM holds the screenplay, per-NPC memory and story state, and mutates the world through tool calls (update_npc, set_story_state); a scene-design agent with six tools (read_screenplay, generate_prop, place_npc, move/delete/scale) running a multi-round agentic loop over a live scene snapshot; a voice pipeline using a local faster-whisper server with push-to-talk arbitration between four consumers of one microphone; and the X2 bridge that streams the player's first-person view to Reactor and displays the restyled video on a floating screen. Props come from Tripo text-to-3D, imported as GLB at runtime with XR grab support. NPC markers are reference-image billboards that are deliberately visible to the video model as conditioning silhouettes, while name labels live on a culled layer so text never pollutes the model input. Everything is code-generated at runtime — the project has almost no prefabs.

Challenges we ran into

On-device WebRTC silently produced zero frames on PICO (encoder format restrictions), which we first fixed with platform-queried encoder textures and finally bypassed with a Mac-side relay bridging headset TCP to Reactor. XRI 3.0's NearFarInteractor turned out not to support uGUI at all, so ray clicks on our panels did nothing — we wrote our own ray-to-rect interaction path with hover, click, drag and corner-resize, then reconciled it with the official pipeline. One microphone had to serve style dictation, NPC conversation, screenplay editing and the scene agent without ever stealing each other's transcripts. Keeping the generative video model's input clean while still showing the player UI required strict camera-layer discipline. And our ngrok tunnel died mid-demo, pushing us to a LAN-direct speech setup.

Accomplishments that we're proud of

A complete closed loop from spoken words to inhabitable, stylized, dramatized world with no editor step in between. A director AI whose story state genuinely persists — walk away from Vera, come back, and she remembers. Physical narrative triggers: handing an object to a character is a story event, not a collision sound. And a six-stage onboarding flow that makes all of this operable from inside the headset alone.

What we learned

Generative pipelines are latency chains — the art is hiding each stage's seconds behind meaningful interaction, like progress signposts planted at the exact spot your prop will appear. LLM agents become reliable when the world is serialized into their context every round and every mutation goes through typed tools. Mobile XR runtimes punish assumptions: formats, layers and input stacks that work in the editor fail silently on device, so every capability needs a fallback path. And voice is a surprisingly complete input device once ownership of the microphone is treated as a first-class design problem.

What's next for SAO

Wiring voice-driven world generation and the world browser directly into the selection stage, so players never see a default world. Multiple simultaneous NPCs with the director orchestrating an ensemble. Hand-tracked interaction to replace controllers. Persistent player-authored worlds and screenplays shared between sessions and between players. And collapsing the relay so the whole loop runs untethered on the headset.

Built With

• ai-agents
• ai-npc
• gaussian-splatting
• generative-ai
• interactive-storytelling
• llm
• openxr
• text-to-3d
• unity
• voice-interface
• vr