Chatter Town

Learn a language by immersing yourself in virtual game world where you can talk, make mistakes, and learn what matters to you.

Comment

Inspiration

I've lived in 7 different countries as an expat, and in every one of them I got anxious when I had to speak the local language.

I tried Duolingo, but it didn't help with that. The lessons follow a script and real people don't, so as soon as someone goes off script you don't know what to say. I also wasted a lot of time on topics that weren't interesting to me.

Chatter Town is my way to practice real conversations. You arrive by bus in a small town and talk out loud with the people who live there, like the barista at the café or the seller at the market. They react to what you actually say, they go off script, and you talk about the things you care about. Lola, the teacher at the school, teaches you a few phrases and then sends you into town to use them. The town is preparing for the Fire, a festival based on Valencia's Fallas, and your last mission is telling your own story to the Keeper in the language you're learning.

How I built it

  • App: Expo / React Native. The town itself is a Godot 4.5.1 game running inside the app, and React draws the conversation on top of it.
  • Voice: every conversation is a live, full-duplex call with OpenAI's GPT-Live over WebRTC, so you can interrupt people and they can interrupt you. A second model runs the town's tools behind the voice: taking your order, charging you, handing over a receipt, making a flashcard. ElevenLabs reads the flashcards out loud.
  • Characters: each person is a short Markdown file with their personality, what they know and which tools they can use. Each place has a real menu and stock, so you can only order what's actually there.
  • Languages: 10 languages. Each town is localised with its own names, prices, currency and the words on the signs, so the Japanese town has a different café menu than the Spanish one.
  • Payments: there are no accounts. The server gives each device an anonymous pass, and that pass is the RevenueCat user. After your first finished mission Lola asks if you accept the challenge of the Fire, and that's where the paywall for Chatter Pro (with a 7-day free trial) comes in.
  • I built most of it with Claude Code, running several agents in parallel on separate parts of the town.

Challenges

  • The voice model never sees its tools. It decides on its own when to hand work to the backend, and it does that about once per turn. Flashcards for phrases Lola had just taught sometimes showed up 30 to 90 seconds late. I fixed this by sending the last lines to the backend myself whenever there's a short pause.
  • Longer prompts made the characters worse. Every extra rule made them sound more scripted, and any example sentence I gave them got repeated word for word. In the end I cut the prompts down to a few lines.
  • Correcting mistakes without making people anxious. Nobody in town says "no" or explains your mistake. They say back what you meant, correctly, and carry on, the way a friendly local would.

What I learned

  • People love taking things off script. The first thing most testers did was flirt with the characters, so the characters need to be able to handle whatever you throw at them.
  • In voice AI, less instruction gives more natural conversation.
  • The way characters correct you here is called a "recast" in language-learning research, and it works best when you change only one thing at a time.
  • Putting the paywall inside the story, after you've had one real conversation, feels much better than showing it at launch.

Built With

Share this project:

Updates

Submission history