Inspiration

"There’s no better way to learn a language than by immersing yourself in a culture where it’s spoken, and there’s no better way to immerse yourself in a culture than by learning to speak the local language." — Wendy Werneth

The fastest way to learn a language has always been to live in it. Most people just never get the chance.

Immersion works. But most people can't move abroad to get it, and apps hand you flashcards instead of a life. We wanted to build the next best thing: a neighborhood you can step into for ten minutes a day, where every errand is a lesson and every neighbor talks back.

What it does

howtohello is a 3D language-learning game. You move into a new neighborhood and learn the way you would if you really lived there: greeting people, unpacking, counting, reading, shopping, and helping your neighbors. Our first language is Nepali, told as a six-chapter story on an illustrated adventure map.

Chapter What you do What you learn
01 · Your First Hello Arrive at your new home, greet your host Asha, collect your room key, and find your suitcase with the porter Greetings, thank you, yes and no: five useful phrases
02 · Unpack Your Suitcase Put your things away by color Five colors
03 · Counting with Lila Help your neighbor fill her basket with apples in a 3D courtyard Numbers one to five
04 · Your First Letters Trace Devanagari letters stroke by stroke, starting at a gold dot, with pencil sounds and each letter pronounced aloud The Nepali vowels (अ आ इ ई उ …), in sets of five
05 · The Little Shop Find your shopping on eight shelves using only direction words, and ask the AI shopkeeper the way Left, right, ahead, behind
06 · The Morning Market Explore a living street market and run errands for three neighbors Everything together: greetings, colors, numbers, directions

Every chapter builds on the last. The colors from Chapter 02 and the numbers from Chapter 03 come back when Maya at the market asks for दुई रातो (two red) apples.

Words that teach through action

English stays hidden unless you ask for a hint. The meaning comes from what happens:

  • The color word paints itself red when you pick a red apple.
  • The number glows when you're holding the right count.
  • Choose अगाडि ("forward") when you meant to say hello, and you actually step forward.

🤖 AI neighbors that understand you

  • Ask the shopkeeper the way. Type "chiya kahaa chha?" ("where is the tea?") and he answers out loud: "चिया तपाईंको दायाँ छ।" ("The tea is on your right.")
  • Talk in your own words. Type to any market neighbor in Nepali, even in Latin letters with typos. The AI decides whether you said the right thing for the moment, and only then does the story move forward.
  • They speak back, each with their own voice and personality.
  • They adapt. Struggling learners get simpler Nepali and the exact word to try; confident learners get a nudge, not the answer.
  • They're never wrong about the world. The game calculates where every shelf really is from where you're standing and hands that to the AI, so it can't send you the wrong way.

Try it in 2 minutes

  1. Open https://howtohello.study, create an account, and choose Nepali.
  2. Play Chapter 01 to meet Asha, or use the ⏭ button in the corner to jump to any chapter.
  3. In Chapter 04, trace your first letter, अ.
  4. In Chapter 05, tap Ask the shopkeeper and type chiya kahaa chha?
  5. In Chapter 06, walk up to Maya, press E, and type namaste.

How we built it

Layer What we used
3D world TypeScript, three.js, Vite: five interactive 3D scenes (home, courtyard, shop, market, and the journey between them)
Letter tracing Canvas stroke tracing with guided start points, coverage checking, pencil audio and per-letter pronunciation
AI conversation OpenAI gpt-4o-mini, structured JSON replies, called from a Vercel server function. Each request carries the character, the lesson step, the learner's history and the game's true directions.
AI voices OpenAI text-to-speech, with a voice and speaking style per character. Lines are cached at Vercel's edge, so repeats are free.
Audio Recorded Nepali clips from multiple speakers, a market soundscape (birds, stall chatter, bicycles), map music, and a shop radio that fades down when someone speaks
Accounts Supabase Auth and Postgres with row-level security, so progress follows you across devices
Hosting Vercel, at howtohello.study
Quality 100+ automated unit and Playwright browser tests that play through the chapters on desktop and phone

The key design decision: the game owns the truth, and the AI owns the conversation. The game decides what each step is and where things are. The AI understands messy learner input and replies naturally at the right level.

Challenges we ran into

  • AI that gives confident wrong directions. We stopped asking it to know the shop and fed it the ground truth instead.
  • Letting the AI judge without letting it skip the lesson. It decides whether "namaste" counts as a greeting, but it can't invent a new step.
  • Checking handwriting fairly. The tracing had to accept wobbly beginner strokes but still require the real letter shape, including the top line Nepali letters hang from.
  • Teaching without walls of text. Our first market drowned in English. We rebuilt it so meaning comes from interaction, not translation.
  • Nepali in a world built for English: Devanagari rendering, toggleable Latin-letter guides, and getting natural speech out of a general voice model.
  • Four people, one weekend, same files. 3D, audio, tracing and AI all collided in the same code, and we merged a lot.

Accomplishments that we're proud of

  • A complete six-chapter journey from your first hello to your first letters to running errands on your own.
  • Someone who has never seen Nepali can finish the shop and market using only Nepali words.
  • AI characters that feel like neighbors, not chatbots, and can't break the lesson.
  • Nothing breaks if the AI does. No key, no internet or no microphone means the buttons and hints simply take over.

What we learned

  • Doing beats reading. A word you act on sticks better than a word you're told.
  • The most useful job for AI here wasn't generating content. It was understanding beginners and answering at their level.
  • Tiny details decide whether beginners feel smart or lost: speaking slowly, shuffling where the right answer appears, and never putting a ✓ next to it.

What's next for howtohello

  • Multiplayer neighborhoods. Explore the same 3D world with real people learning the same language as you. Walk up to another learner and talk through proximity voice chat, the way you would in a real market: the closer you are, the louder they get.
  • AI neighbors in a shared world. AI shopkeepers and townspeople keep living around you while you play with others. They start conversations, give you errands to solve together, and react to what you and your friends say. Practice happens with humans and AI at the same time.
  • An AI helper that whispers. Freeze mid-conversation with another learner? A quiet AI helper suggests what to say next, at your level, so you never get stuck.
  • More neighborhoods, more languages. Spanish, Hindi and Japanese, with matchmaking that pairs you with learners of the same language and a similar level.
  • Speak instead of type, with pronunciation feedback.
  • The full alphabet: consonants and joined letters after the vowels.
  • Families and classrooms: progress views for parents, and private class neighborhoods where teachers and students can practice together.

Built With

Share this project:

Updates

Submission history