Inspiration

After watching Cyberpunk: Edgerunners season 2, I was struck by Project Capricorn, an AI that can hack into things and control them. I wanted to build a version for my own Quest: a spatial assistant that controls my smart devices, uses spatial data to work out which device I mean, looks like it is hacking into things, and can get real work done when I ask.

What it does

Capri is a spatial AI companion for Meta Quest. She is an animated avatar who stands on your real floor in passthrough AR. Point your right index finger at a light, hold a left-hand pinch, and say “turn off that.” Capri looks at the lamp, flicks her hand, and sends beams, glyphs and sparks to it. The command fires when the effect lands, and she reports the real result. On my own Quest 3S I controlled a real smart bulb this way: on, off, brightness and colour, just by pointing and speaking. She also fidgets, glances at your furniture, and teases you if you toggle the same lamp repeatedly. The experience is hands-first and seated. The hosted link also includes a Mock Mode with virtual bulbs, to try out Capri's capabilities before connecting to actual devices.

How we built it

Capri is a static IWSDK (WebXR) app with no backend. Three.js and three-vrm render the VRM avatar, which a procedural animator drives. She uses hand tracking, plane and mesh detection, hit-testing and spatial anchors. Speech recognition runs on the headset with Vosk and a restricted grammar, so no audio leaves the device. Her voice is a library of 58 pre-generated Piper lines, because Quest Browser has no speech synthesis. Intent parsing, pointing and personality logic are pure TypeScript with unit and property tests.

The agentic layer is optional and bring-your-own. Users can connect their own Home Assistant for real lights and their own self-hosted Hermes Agent for free-form questions and long-running agentic tasks. Long tasks appear on a task panel with live steps and Approve and Deny buttons. Credentials stay on the headset and go only to the endpoints the user enters.

Challenges we ran into

A microphone prompt inside AR ends the session, so permission is requested on a 2D page first. Since the Quest Browser has no speech synthesis, speech recognition runs on the headset with a restricted grammar. The voice grammar depends on word order, so commands like “turn that off” work best. Dynamic replies appear as speech bubbles rather than speech.

Accomplishments that we're proud of

It works on a real light. On my Quest 3S I turned a real smart bulb on and off and changed its brightness and colour just by pointing and talking. That moment was the whole reason I started the project. Pointing plus voice resolves “that.” Capri works out which device I mean from where my finger points, so “turn off that” works without naming anything. Fully hands-only. The entire experience, setup included, works without a controller. Eight poke buttons and a hand keyboard cover every voice command. No backend, and private by design. Capri is static files. Speech recognition runs on the headset, and credentials never leave the device except to the user’s own endpoints. She never claims something she didn’t do. A light result comes from the real response, a task’s closing line comes from its real final state, and an agent action runs only if its target exists in the room. Anyone can try it. The hosted link opens in Mock Mode with virtual bulbs, so no smart home is needed. A first VR project. I built it as a solo developer with no prior VR experience, and it’s already published.

What we learned

Mixed reality has platform quirks. A microphone prompt inside an AR session ends the session, so permission has to be requested on a 2D page first. Quest Browser also has no speech synthesis, which is why her voice is a pre-generated library. Constraints made it better. A restricted voice grammar made on-device recognition accurate and fast, and it kept the experience reliable. Pointing needs generous design. Fingers drift, so I latch the pointed light when push-to-talk starts and use large hit volumes. Spatial context is what makes it feel smart. Knowing where things are did more for the experience than any language feature. Hand tracking is ready for things that used to be sci-fi. The mix of pointing, speech and a character reacting in your room felt magical.

What's next for Project Capri

I’ll keep developing Capri. The biggest step is connecting an LLM so she can hold real-time conversations in her own persona, with dynamic responses instead of preset ones. After that come a custom avatar and voice, and more device types.

How hand interactions are used

CAPRI is designed around hands-first interaction. Users point with their right index finger to select a spatial object and use pinch gestures to confirm interactions. A left-hand pinch activates push-to-talk, allowing users to issue voice commands while keeping their controllers aside. Spatial UI elements also support direct hand-based poking.

Built With

Share this project:

Updates

Submission history