Phone home screen: Start a conversation, set up a cloned voice, or manage saved phrases.
Voice consent: Cloning only starts after the speaker confirms permission.
Record the voice: About 30 seconds of reading a short passage is enough for Qwen3-TTS to clone it locally.
Voice ready: Cloning finishes locally, and every reply from now on is spoken in the user’s own voice.
Saved phrases: Common phrases are grouped by place, like Hospital or Home, and play in the user’s own voice from the headset in one action.
Listening: The phone streams the conversation to local speech-to-text, and live captions appear as the other person speaks.
Headset view: The partner’s question appears as a live caption, with three AI-suggested replies floating beside them
Headset menu: Recenter the view, move the listener anchor, browse past conversations and memories, or adjust settings
Saved phrases in the headset: Phrases are grouped by place (Doctor, Home, Family, Out and about)
Saved conversations: Tag a talk by who it was with, then attach a past conversation so suggestions pick up where you left off
Memories: Facts saved with “Remember this,” like an upcoming appointment.
Settings: Choose hands, head, or both for selection, tune the gaze hold time from 0.6 to 1.6 seconds.
Type my own: When no suggestion fits, users build a reply from common words by gaze

Inspiration

People who can't speak need to keep eye contact while they communicate. Eye contact is how a conversation stays alive: it shows you're present, that you heard, and that an answer is coming. Today's augmentative and alternative communication (AAC) tools ask users to look down at a tablet or letter board. While they type, the other person waits, the moment passes, and long silences get mistaken for "done."

This affects people after a stroke, a laryngectomy, or vocal-cord paralysis. Many of them understand everything and know exactly what they want to say. What's missing is a way to say it at the speed of a conversation, in a voice that sounds like them, without turning away from the person in front of them.

What it does

Cadence lets someone who can't speak (stroke/aphasia, laryngectomy, vocal-cord paralysis) wear a Meta Quest 3S in pass through, see the person they're talking to, and respond by picking from AI-suggested replies floating beside them. A an attached mobile app picks up the conversation, suggests three short replies fitting their style, and speaks the chosen one in their own cloned voice. Selection works by hand motions or head movement. Instant gestures say Yes/No/Give me a second in under a second.

How we built it

Cadence was built as a local-first, distributed system with a strict focus on low latency, accessibility, and user consent. We approached it by designing the protocol first, then building each component around it.

1. Protocol-First Architecture

Before writing any UI or model code, we established a single source of truth for communication:

Typed Contracts: We defined every event in JSON Schema (packages/protocol/schema/), and generated TypeScript types and Go structs from them. This allowed our headset, phone, and gateway to all share the exact same types.
Contract Testing: Every schema has an example event in packages/protocol/examples/, and our CI tests parse all of them on both the TypeScript and Go sides to prevent drift.
Event-Driven Design: All communication goes through a WebSocket hub with a strict envelope (type, v, sessionId, turnId, ts, data). Phone microphone audio is the only exception, sent as raw PCM16 16kHz binary frames.

2. The Gateway (Orchestration & State)

The Go gateway (services/gateway) is the heart of the system and the only process exposed to the network.

State Machine: We implemented a strict session state machine (IDLE → LISTENING → SUGGESTING → CHOOSING → COMPOSING → SPEAKING) that broadcasts transitions to both clients to keep them in sync.
Speculative Synthesis: This was our biggest latency win. While a person is still speaking, we debounce transcript.partial events (~400ms) and call the suggestion model. For the top 3 replies, we pre-synthesize their TTS in the background and cache the clips. When the user selects a reply, it's almost always already generated.
Intelligent Caching: Clips are keyed by (voiceId, locale, text), so repeated phrases (like "Yes" or saved phrases) are never synthesized twice.
Local-Only Data: We used modernc.org/sqlite to store consent, voice profiles, saved phrases, audit logs, and metrics. Conversation content is only ever written to disk if explicit store_chat consent is granted.

3. AI Pipeline (Local & Swappable)

Our Python FastAPI service (services/models) runs entirely on localhost with swappable backends to support both our M4 Pro and a potential CUDA server.

STT: Streaming speech-to-text with mlx-whisper, chunking audio and emitting partial and final transcripts over WebSocket.
Suggestions: We prompt Gemma 4 E4B (via MLX or Ollama) to return exactly 3 short, conversational replies conditioned on history, user style, and context. We heavily constrain the output to JSON to ensure reliability.
Voice Cloning & TTS: Using Qwen3-TTS via mlx-audio, we implemented a consent-gated voice creation flow. The service can transcribe samples itself, extract audio from uploaded videos with ffmpeg, cache voice prompts, and pre-synthesize phrases for near-instant playback.
Resiliency: Every model call has timeouts (3s for suggestions, 6s for TTS), retries on JSON parse failures, and the gateway is designed to gracefully degrade if the model service crashes (allowing instant phrases and typing to still work).

4. XR Headset & Phone Clients

We wanted this to feel as seamless as possible for both the user and the person they're talking to.

Headset (Quest 3S): Built with Vite, React, TypeScript, React Three Fiber, and @react-three/xr in immersive-ar passthrough mode. We render floating reply bubbles anchored in 3D space, implemented hand tracking for pinch/poke, custom gesture recognition for thumbs-up/down/open palm, and a head-gaze dwell reticle with a filling ring for hands-free accessibility.
Phone: A lightweight web app that captures microphone audio via getUserMedia and an AudioWorklet that downsamples to PCM16 16kHz to send as 20ms binary chunks. It uses the Web Audio API to play synthesized clips with precise timing, displays live captions, and shows an "I'm answering" indicator to signal to the conversation partner that a response is coming.
Shared Logic: apps/shared contains our WebSocket client with exponential backoff, a shared Zustand store, and locale files to allow for future internationalization.

5. Tooling, DX, & Local-First Constraints

Monorepo: Managed with pnpm workspaces, Go modules, and uv for Python dependencies, all orchestrated by a unified Makefile.
Local Development: We use mkcert for trusted HTTPS on the LAN (required by getUserMedia and WebXR), and built mock servers (mock-gateway and mock-models) to allow frontend and backend work to be done in parallel.
Safety by Design: Consent is required before cloning voices, every played utterance is written to an audit log, revoking consent cascades to delete all associated data, and .env is used for all configuration with nothing hardcoded.

Challenges we ran into

Latency on Local Hardware: Running STT, Gemma 4, and Qwen3-TTS on a 24GB M4 Pro was tight on memory. We used speculative synthesis (pre-generating TTS for the top 3 replies) and aggressive clip caching to keep responses fast.
WebXR Limitations: Quest Browser lacks passthrough camera access (blocking object pointing), has reserved OS gestures that clash with ours, and its emulator can't test hand tracking, forcing most testing on-device.
Accessibility Tuning: Finding the right head-dwell time, movement tolerance, and feedback took lots of iteration to prevent accidental selections without slowing down conversation.
Cross-Device Audio & Networking: Browser autoplay policies, 20ms PCM16 mic streaming, and trusting mkcert LAN certificates across Quest, iOS, and Android were all incredibly finicky to get working.
Privacy, Sync & Reliability: We enforced consent-gated cloning with full deletion on revoke, kept TS and Go in sync via schema-generated types, added turnId-based cancellation for stale audio, and made the gateway gracefully degrade if the local model service crashes.

Accomplishments that we're proud of

What we learned

Perceived latency matters most: Speculative synthesis (pre-generating TTS while someone speaks) made conversations feel natural far more than optimizing raw model speed.
Local ML is viable: We proved STT, suggestions, and voice cloning can all run privately on a single laptop using MLX with smart caching.
WebXR is restrictive: Quest Browser's limitations (no passthrough camera access, reserved gestures, poor emulation) forced heavy on-device testing and early pivots.
Shared contracts save time: Generating Go and TypeScript types from a single JSON Schema prevented constant protocol drift across our stack.
Accessibility takes iteration: Tuning head dwell, movement tolerance, and gesture detection was crucial to balance speed with avoiding accidental selections

What's next for Cadence

Saved conversations & memories: Add consent-gated context from past visits to make suggestions smarter and more personalized.
Object pointing: Revisit "point and ask" functionality if Quest Browser exposes passthrough camera access in the future.
Expanded input modes: Add head gestures (nod/shake) and customizable dwell profiles for a wider range of motor abilities.
Real-world testing: Work with members of the stroke, aphasia, and laryngectomy communities to refine Cadence for everyday use.