LingoVille

Inspiration

English isn’t the first language for any of us, but today we all speak it every day. None of us got there through a class or an app. We got there through games: trading items, teaming up with strangers, figuring out what other players meant and making ourselves understood. For me, it clicked at about 11, in a Roblox roleplay game, when I walked my character into a grocery store, talked to the cashier, and got my food without a single hiccup. It was a tiny moment, but it was the first time English felt like something I could actually use.

Plenty of people have a long streak in a drills app and still freeze when it’s time to speak. What’s missing is a safe place to use the language and be understood. So the three of us built a life sim where you eat, stay healthy, and get through your day in English, Chinese, German, or Japanese. Because the best way to learn a language is to live in it.

What it does

LingoVille is a 3D life sim in the browser. It's set in a town whose people only understand your target language (Japanese, Chinese, English or German).

Walk up to anyone, press E, hold Space and speak to order coffee, pay rent, see a doctor or join the gym.
Work café Shifts where customers order out loud and your pay depends on how many orders you got right.
Keep your Well-being and Mood up, and pay rent each week before it turns into debt.
NPCs quietly adapt to a hidden proficiency level (A1–C2). Stuck? Use Help for a hint, and Translate or Replay any line.
Every conversation ends with a Recap of up to three corrections.

How we built it

Frontend: TypeScript, React, React Three Fiber and Rapier for the 3D town and physics, Zustand for state, and Vite.
Voice: the Gemini Live API streams your mic (downsampled to 16 kHz PCM) and plays the NPC's reply, each NPC in their own voice.
Gateway: a small Node server holds the Gemini key, hands out one-use session tokens, and writes the hints, translations and Recaps.
Art: buildings and furniture modelled in Blender and exported as GLB.
Process: spec-first and test-first with AI coding agents. Every feature started as a spec and tickets and was built red→green.

Challenges we ran into

Making NPCs actually refuse to understand you when you're unclear, without becoming frustrating.
Real-time voice latency and audio handling in the browser.
Keeping a growing 3D town running smoothly across different graphics cards.
Testing a voice game reliably, which needed a mock mode with a scripted NPC so it runs without a key or any cost.

Accomplishments that we're proud of

A full daily-life loop where every step is a spoken conversation.
Fifteen named NPCs with their own personalities, voices and memory of you.
Barista, cashier and server jobs, plus clinic, restaurant, bookshop, gym and rent systems.

What we learned

How to work with AI agents: write the spec first, keep tickets small, and let tests decide when something is done.
Prompting a live voice model to stay in character and at the right language level is its own design problem.