Sundai
An AI you don't just talk to. An AI you share a world with.
Team Simulabs | Stanford Immerse the Bay 2026
Sundai is an embodied AI companion who sees, speaks, moves, remembers, and interacts with the world around you. Meet her in your own room, walk beside her, or challenge her to Swords and Arrows, an immersive AR/VR strategy game where chess meets medieval combat.
We built Sundai around one belief: The future of AI isn't another voice trapped behind a screen. It's an intelligence that can experience the world alongside you.
Inspiration
Every AI we use today lives behind a screen. We ask questions, receive answers, and close the application. But something is missing.
When you spend time with another person, you don't just exchange words. You make eye contact, walk together, notice things, and share experiences.
We wondered: What would happen if we gave AI a body?
Not just a 3D avatar attached to a chatbot, but an intelligent companion that could understand its environment, make decisions, move independently, and participate in your world.
We wanted someone you could call over, walk with, talk to, and challenge to a game. Someone who could be present rather than simply responsive.
And we wanted to demonstrate why that presence matters.
That's how Sundai began.
What it does
Put on a Meta Quest headset, open our WebXR experience, and Sundai appears in your room at human scale.
She talks with you. Her real-time conversational AI allows natural back-and-forth dialogue, including interruptions, without the long pauses of traditional voice assistants.
She moves with you. Ask her to come closer, stand beside you, follow you, sit down, or walk somewhere. She can also wander independently. Her movement system handles navigation, orientation, personal space, and natural transitions between behaviors.
She feels present. Sundai uses 17 motion-capture animations, natural head and eye movement, expressive gestures, and mouth animation synchronized with her voice. She looks toward you when speaking and moves with an awareness of where you are.
She sees your surroundings. Using local computer vision, Sundai can identify everyday objects through the headset's outward-facing camera and remember where she last observed them. Recognition runs directly in the browser, without sending camera images to the cloud.
She plays with you. Ask Sundai to play a game, and your room transforms into a castle terrace at sunset. She changes into a medieval court gown, a strategy board appears between you, and she becomes your opponent and coach.
Swords and Arrows — Fight for the square
What if chess pieces didn't simply capture one another? What if you actually had to fight for every square?
Swords and Arrows is an original 5×5 strategy game where each player controls archers, swordsmen, and a monarch.
But when one piece attacks another, the experience transforms from an AR board game into a first-person VR battle.
- Sword combat: Defend yourself as enemy soldiers charge in waves.
- Arrow deflection: Physically knock incoming arrows out of the air with your sword, sending them back toward enemy archers.
- Archery: Hold a bow in one hand, draw the string with the other, and fire at charging or evasive enemies.
Win the duel, and your piece captures the square. Lose, and your attacking piece is eliminated. Capture the opposing monarch to win the game.
The battlefield comes alive with medieval ruins, fires, smoke, catapults, weapons, cinematic lighting, and spatial audio.
Your physical skill becomes part of your strategy. You might make the perfect move on the board, but if you can't win the battle, the strategy falls apart.
And throughout it all, Sundai is there.
An AI who understands the game
Sundai isn't following a scripted conversation or playing random moves.
We built a custom game intelligence system that searches possible moves, evaluates tactical positions, and adapts to how well you've performed in previous battles.
She understands the rules, the current board, and the outcomes of combat. You can ask why she made a move, request a hint, or ask her to explain how a particular battle works.
The game engine handles legal moves and strategy, while her conversational AI interprets the experience and talks with you naturally.
When you're finished, simply ask to leave the game.
The castle disappears. You're back in your room.
The game ends. The companion remains.
How we built it
We combined conversational AI, embodied intelligence, computer vision, spatial computing, and real-time game development into one browser-based experience.
Voice and intelligence: Sundai uses an ElevenLabs conversational agent with Claude as the underlying language model, connected through a real-time audio interface. We developed an executable skill system that allows the AI to turn its decisions into actions in the XR environment.
These skills include moving toward the user, following, sitting, walking to locations, exploring, recognizing objects, and starting or leaving games. When Sundai decides to act, the browser executes the corresponding action.
Body and movement: We used a rigged human avatar with 17 motion-capture animation clips. A custom animation layer handles transitions between walking, speaking, listening, and other behaviors, while her head, eyes, and mouth respond to the conversation. Her movement system plans routes, respects personal space, and rotates her naturally before moving.
Vision and memory: TensorFlow.js and COCO-SSD process camera frames locally to recognize common objects. Sundai stores approximate directions and recent observations, allowing her to answer questions about what she sees and where objects were last observed.
Game intelligence: We created a three-ply search engine that evaluates legal moves and incorporates the player's previous success in different combat matchups. Sundai receives the actual game state and tactical evaluations, allowing her to provide grounded coaching and commentary.
XR gameplay: Built with WebXR, Three.js, and JavaScript, Swords and Arrows combines an AR strategy board with fully immersive VR battles. We developed two-handed archery, sword combat, arrow deflection, enemy behavior, and seamless transitions between the physical room and virtual battlefield.
The experience runs from a URL using GitHub Pages, without requiring a traditional application installation.
Challenges we ran into
Making AI conversation feel natural. Our original speech-to-text, reasoning, and text-to-speech pipeline introduced noticeable delays. Moving to a streaming conversational agent significantly improved responsiveness and allowed natural interruptions.
Giving an AI believable movement. Early versions of Sundai slid sideways, turned mechanically, or moved too close to the player. We developed movement rules that mirror natural behavior: face your destination, maintain personal space, make eye contact, and transition smoothly between actions.
Connecting reasoning to reality. A language model can say "I'm coming over" without actually moving. We connected Sundai's decisions to executable tools so that her words and physical actions remain connected. We also separated the conversational AI from the game engine to keep moves legal and responses grounded.
Moving seamlessly between AR and VR. Switching XR sessions initially interrupted gameplay and required additional user interactions. We redesigned the experience to render the castle and battlefield within a continuous immersive session.
Optimizing for standalone hardware. Detailed characters, animations, effects, and physical combat required careful optimization to keep the experience responsive on Quest hardware.
Accomplishments that we're proud of
Our biggest accomplishment is bringing everything together into a working headset experience.
In one continuous interaction, you can meet Sundai in your room, have a conversation, watch her move around you, ask her to play a game, enter a medieval kingdom, fight a battle with your own hands, and return to the same companion.
We didn't just build an AI chatbot or an XR game. We built a system where intelligence, embodiment, and gameplay interact with one another.
Sundai can understand the game, react to what's happening, and participate in an experience with the player.
And our three-person team brought it together over a single hackathon weekend.
Meet Team Simulabs
Colson — AI Systems, Embodiment & Integration
Colson built Sundai's AI system, including real-time voice and reasoning, the executable skill architecture, and the systems connecting AI decisions to physical actions. He developed her motion-capture animation integration, natural movement, navigation, gaze behavior, and character interactions.
He also built her local computer vision and object-memory capabilities, allowing Sundai to recognize and remember things in the user's surroundings. For Swords and Arrows, Colson developed the adaptive game-search engine and strategic coaching system, connecting the live board state to Sundai's conversational intelligence. He also contributed to the castle environment and integrated Sundai across the experience.
Arthur — WebXR Gameplay & Combat Engineering
Arthur brainstormed game ideas, worked on the WebXR prototype, and implemented game mechanics. That covers the headset experience for the board game: positioning and sitting at the table, moving and fighting in mixed reality, and establishing context while talking with Sim while you play the game. I kept it playable on a Quest and published it so the team could try each round on the device
Tanaya — 3D Art, Environment Design & XR Optimization
Tanaya created and refined the game's chess-inspired characters, transforming initial low-poly models into detailed medieval pieces. She developed battlefield assets and environments featuring ruins, fires, smoke, catapults, weapons, and cinematic lighting, while optimizing graphics and interactions for smooth gameplay on Quest hardware.
What we learned
We learned that presence comes from the smallest details. Eye contact, conversational distance, and natural movement can make a virtual character feel remarkably different from a traditional assistant.
We also learned that embodied AI works best when language models handle reasoning and conversation while deterministic systems handle movement, physics, and game rules.
Most importantly, we learned that the real opportunity in mixed reality isn't simply placing digital objects into physical spaces.
It's creating experiences where people and intelligent virtual characters can actually do things together.
What's next for Simulabs
Swords and Arrows is just one example of what an embodied AI companion can do.
Our vision extends far beyond games.
We want Sundai to understand rooms and objects more deeply, develop reusable skills, remember meaningful experiences, and help people with everyday activities.
Imagine an AI companion who can help you find something you've misplaced, guide you through a difficult task, explore a new place with you, or simply spend time playing and learning alongside you.
We also want to develop more realistic movement, richer personalities, improved spatial awareness, and support for lightweight AR glasses.
Our long-term vision for Simulabs is to create intelligent virtual companions that can exist naturally alongside us, moving between experiences without losing their personality, memory, or relationship with the user.
Sundai is only our first step.
For decades, we've been working toward AI that can understand the world.
We believe the next step is AI that can participate in it.
And perhaps one day, interacting with artificial intelligence won't feel like using a computer at all.
It will feel like sharing an experience.
Built with
WebXR · Three.js · JavaScript · TypeScript · Meta Quest · ElevenLabs · Claude · TensorFlow.js · Node.js · Vite · GitHub Pages · WebRTC
Sundai — An intelligence you don't just talk to. An intelligence you share a world with.
Built With
- chatgpt
- claude
- elevenlabs
- github
- javascript
- meshy
- meta
- node.js
- quest
- tensorflow.js
- three.js
- typescript
- vite
- webxr