Speak Your Way

Most language apps teach you to recognize a word on a flashcard. Then you land in a real city, someone answers your carefully rehearsed sentence with something you didn't expect, and the flashcards stop helping. The hard part is the loop: read a sign, ask a stranger, understand their reply, and do something with it.

We wanted a way to practice that loop. Speak Your Way drops you into a Shanghai-style city where the only way to get things done is to speak Mandarin to the people who live there. You order a drink, ask for directions to 上海大酒店, and get a passerby to take your photo by the 黄浦江. Three of the four people on our team were at their first hackathon, so we also wanted a build process that let beginners work in parallel without breaking each other's work.

What it does

A streamed 3D city. You control a third-person character and walk through streets of lane houses, shops, parks and creeks to a Huangpu River waterfront. The world streams in chunks as you move, and the chunk edges line up without visible seams.
Spoken Mandarin conversations. Your microphone captures speech-to-text, an AI model writes the NPC's reply, and you hear that reply spoken in Mandarin. For accessibility, typed input and subtitles work in the same world.
Three quests: a. Buy a drink. The vendor greets you with 你想喝什么？ and then tells you the coffee is sold out (咖啡卖完了), so you have to adapt and pay for tea or water. b. Ask for directions to 上海大酒店. The route the NPC gives you is computed from the city's real street graph. c. Take a photo at 黄浦江. You ask a passerby to stand on the spot. After the NPC counts down, you get an in-game photo.
Adaptive practice. Each NPC has patience and actions use up patience. The game tracks which skills you have shown (reading, listening, conversation and repair), then adjusts the next challenge to match.

How we built it

Stack: A TypeScript monorepo with three parts: a React Three Fiber client, a Node server (apps/server) and a shared package (packages/shared). The shared package defines contracts and the city logic. We tested with Vitest and Playwright.

AI providers:

Gemini Live handles real-time voice. It runs one two-way audio session with voice-activity detection and barge-in.
Claude handles text dialogue with structured proposals to compose city neighborhoods, but only when a feature flag is turned on.
ElevenLabs provides Mandarin speech-to-text and pre-recorded clip paths. We used only voices that a person on our team had listened to and approved.

The core rule: AI proposes, deterministic code decides. The AI interprets what you said and writes dialogue. The server owns every game fact: prices, inventory, routes and whether a quest is complete. An NPC saying "here's your tea!" never puts tea in your inventory. Only a purchase the server has validated does that.

City generation. A seeded planner lays out the districts, roads and rivers first, so roads and riverbanks line up across chunks no matter which chunk loads first. Claude may propose the neighborhood, but it can only pick from an approved catalog of more than 60 reusable components: storefronts, street pieces, landscaping, waterfront pieces and interaction points. Deterministic code then checks every asset ID and parameter range, and checks for collisions, rendering-budget overruns and blocked paths. If Claude is slow, refuses or returns something invalid, the game immediately falls back to a seeded chunk, so it never waits on a model.

Truthful directions. For Quest 2, the route uses the city's actual road segments. Each turn adds a penalty to the route's cost, so the directions stay short and easy to say out loud. The NPC can only name streets and landmarks that already exist in the loaded city.

Gentle quest bias. Suppose you haven't come across a place in a while, like a drink vendor, a helpful local or a river viewpoint. The city then becomes slightly more likely to place one. The boost is capped and balanced against the other categories, so the city learns without being rigged. Separately, a deterministic placement step makes sure every quest goal can be reached within a set walking distance.

Adaptive difficulty. Only evidence you produce on your own, on your first attempt, counts toward your skill tracking. Subtitle-assisted listening and edited transcripts don't. The game raises the difficulty only after several independent successes and lowers it when you're consistently struggling.

How four people built in parallel. We split the work into numbered, self-contained "segments." Each segment had its own files, frozen interface contracts and acceptance criteria. Anyone on the team could say "Build Segment-003" in a fresh session on their own machine, and that segment would be implemented, tested and merged without colliding with anyone else's work.

Challenges we ran into

Voice latency versus voice consistency. Gemini Live gave us real-time conversation with interruption, but only in its own built-in voices. We tried sending Gemini's replies through ElevenLabs so every NPC would use a voice we had approved. That added about 1–2 seconds because synthesis can't start until the full reply text has arrived. We kept the faster live path and documented the voice gap as future work.
Measured latency. In live runs, the time from speech to the start of the NPC's reply audio was roughly 2–5.5 seconds, and almost all of that was the model generating its response. Getting the first audio to play sooner became its own optimization pass.
Streaming a world that has to stay true. Chunks load in whatever order the player walks, yet the roads have to connect and the hotel can never move. If a chunk generation fails or arrives late, it is discarded so it never replaces a chunk the player has already seen. The chunks around the spawn point are pinned, so walking back there never triggers regeneration.

What we learned

Treat the LLM as an interpreter, not a referee. When they make proposals and code makes the decisions, whole categories of bugs disappear: free items, made-up roads and quests that completed just because the AI said so.
Freeze contracts before you write code when working in parallel. Locking down the interfaces between segments early saved us hours of merge conflicts.
Always build the fallback. The seeded city generator and the fixture dialogue providers let us demo, test and run CI without network access or API keys.
Speed is part of the language lesson. A 5-second pause breaks the feeling of a real conversation more than an imperfect reply does.

What's next

Use our approved ElevenLabs voices in live runs without extra latency. We could switch Gemini Live to text-only output or build our own streaming pipeline that runs speech-to-text, then the LLM, then text-to-speech.
Add more quests built on the same rule: the server decides state, not the AI.