Inspiration
"There’s no better way to learn a language than by immersing yourself in a culture where it’s spoken, and there’s no better way to immerse yourself in a culture than by learning to speak the local language." — Wendy Werneth
The fastest way to learn a language has always been to live in it. Most people just never get the chance.
Immersion works. But most people can't move abroad to get it, and apps hand you flashcards instead of a life. We wanted to build the next best thing: a neighborhood you can step into for ten minutes a day, where every errand is a lesson and every neighbor talks back.
What it does
howtohello is a 3D language-learning game. You move into a new neighborhood and learn the way you would if you really lived there: greeting people, unpacking, counting, reading, shopping, and helping your neighbors. Our first language is Nepali, told as a six-chapter story on an illustrated adventure map.
| Chapter | What you do | What you learn |
|---|---|---|
| 01 · Your First Hello | Arrive at your new home, greet your host Asha, collect your room key, and find your suitcase with the porter | Greetings, thank you, yes and no: five useful phrases |
| 02 · Unpack Your Suitcase | Put your things away by color | Five colors |
| 03 · Counting with Lila | Help your neighbor fill her basket with apples in a 3D courtyard | Numbers one to five |
| 04 · Your First Letters | Trace Devanagari letters stroke by stroke, starting at a gold dot, with pencil sounds and each letter pronounced aloud | The Nepali vowels (अ आ इ ई उ …), in sets of five |
| 05 · The Little Shop | Find your shopping on eight shelves using only direction words, and ask the AI shopkeeper the way | Left, right, ahead, behind |
| 06 · The Morning Market | Explore a living street market and run errands for three neighbors | Everything together: greetings, colors, numbers, directions |
Every chapter builds on the last. The colors from Chapter 02 and the numbers from Chapter 03 come back when Maya at the market asks for दुई रातो (two red) apples.
Words that teach through action
English stays hidden unless you ask for a hint. The meaning comes from what happens:
- The color word paints itself red when you pick a red apple.
- The number glows when you're holding the right count.
- Choose अगाडि ("forward") when you meant to say hello, and you actually step forward.
🤖 AI neighbors that understand you
- Ask the shopkeeper the way. Type "chiya kahaa chha?" ("where is the tea?") and he answers out loud: "चिया तपाईंको दायाँ छ।" ("The tea is on your right.")
- Talk in your own words. Type to any market neighbor in Nepali, even in Latin letters with typos. The AI decides whether you said the right thing for the moment, and only then does the story move forward.
- They speak back, each with their own voice and personality.
- They adapt. Struggling learners get simpler Nepali and the exact word to try; confident learners get a nudge, not the answer.
- They're never wrong about the world. The game calculates where every shelf really is from where you're standing and hands that to the AI, so it can't send you the wrong way.
Try it in 2 minutes
- Open https://howtohello.study, create an account, and choose Nepali.
- Play Chapter 01 to meet Asha, or use the ⏭ button in the corner to jump to any chapter.
- In Chapter 04, trace your first letter, अ.
- In Chapter 05, tap Ask the shopkeeper and type
chiya kahaa chha? - In Chapter 06, walk up to Maya, press E, and type
namaste.
How we built it
| Layer | What we used |
|---|---|
| 3D world | TypeScript, three.js, Vite: five interactive 3D scenes (home, courtyard, shop, market, and the journey between them) |
| Letter tracing | Canvas stroke tracing with guided start points, coverage checking, pencil audio and per-letter pronunciation |
| AI conversation | OpenAI gpt-4o-mini, structured JSON replies, called from a Vercel server function. Each request carries the character, the lesson step, the learner's history and the game's true directions. |
| AI voices | OpenAI text-to-speech, with a voice and speaking style per character. Lines are cached at Vercel's edge, so repeats are free. |
| Audio | Recorded Nepali clips from multiple speakers, a market soundscape (birds, stall chatter, bicycles), map music, and a shop radio that fades down when someone speaks |
| Accounts | Supabase Auth and Postgres with row-level security, so progress follows you across devices |
| Hosting | Vercel, at howtohello.study |
| Quality | 100+ automated unit and Playwright browser tests that play through the chapters on desktop and phone |
The key design decision: the game owns the truth, and the AI owns the conversation. The game decides what each step is and where things are. The AI understands messy learner input and replies naturally at the right level.
Challenges we ran into
- AI that gives confident wrong directions. We stopped asking it to know the shop and fed it the ground truth instead.
- Letting the AI judge without letting it skip the lesson. It decides whether "namaste" counts as a greeting, but it can't invent a new step.
- Checking handwriting fairly. The tracing had to accept wobbly beginner strokes but still require the real letter shape, including the top line Nepali letters hang from.
- Teaching without walls of text. Our first market drowned in English. We rebuilt it so meaning comes from interaction, not translation.
- Nepali in a world built for English: Devanagari rendering, toggleable Latin-letter guides, and getting natural speech out of a general voice model.
- Four people, one weekend, same files. 3D, audio, tracing and AI all collided in the same code, and we merged a lot.
Accomplishments that we're proud of
- A complete six-chapter journey from your first hello to your first letters to running errands on your own.
- Someone who has never seen Nepali can finish the shop and market using only Nepali words.
- AI characters that feel like neighbors, not chatbots, and can't break the lesson.
- Nothing breaks if the AI does. No key, no internet or no microphone means the buttons and hints simply take over.
What we learned
- Doing beats reading. A word you act on sticks better than a word you're told.
- The most useful job for AI here wasn't generating content. It was understanding beginners and answering at their level.
- Tiny details decide whether beginners feel smart or lost: speaking slowly, shuffling where the right answer appears, and never putting a ✓ next to it.
What's next for howtohello
- Multiplayer neighborhoods. Explore the same 3D world with real people learning the same language as you. Walk up to another learner and talk through proximity voice chat, the way you would in a real market: the closer you are, the louder they get.
- AI neighbors in a shared world. AI shopkeepers and townspeople keep living around you while you play with others. They start conversations, give you errands to solve together, and react to what you and your friends say. Practice happens with humans and AI at the same time.
- An AI helper that whispers. Freeze mid-conversation with another learner? A quiet AI helper suggests what to say next, at your level, so you never get stuck.
- More neighborhoods, more languages. Spanish, Hindi and Japanese, with matchmaking that pairs you with learners of the same language and a similar level.
- Speak instead of type, with pronunciation feedback.
- The full alphabet: consonants and joined letters after the vowels.
- Families and classrooms: progress views for parents, and private class neighborhoods where teachers and students can practice together.
Built With
- canvas
- css3
- github
- gpt-4o-mini
- html5
- javascript
- node.js
- openai
- openai-text-to-speech
- playwright
- postgresql
- serverless
- supabase
- three.js
- typescript
- vercel
- vite
- web-audio-api
- webgl
