Babbli
Built at HackTheHill III, Babbli won four of the event's 21 prizes:
- 🥉 General Challenge — Third Place
- 🎙️ Best Project Built with ElevenLabs — Best Use of ElevenLabs
- 📚 Best Educational Project — MathemaTech: Education for Everyone
- 🎨 Best UI/UX
Learning a language through vocabulary lists and scripted exercises is very different from actually using it in the real world.
You might know how to order food in French on paper, but when a waiter speaks quickly, asks an unexpected follow-up question, or does not understand you the first time, the experience becomes completely different.
We wanted to recreate that pressure in a simulated but fun environment.
Babbli lets you practice real conversations before you have to live them.
Instead of chatting with a generic AI tutor, learners are placed inside first-person simulations where they must accomplish a real objective by speaking or writing in the target language.
What it does
Babbli is a gamified first-person language practice simulator powered by AI characters.
A learner chooses a language, difficulty level, response mode, and scenario, then enters an immersive environment such as:
- shopping for and buying a specific item at a department store in New York
- ordering ramen at a restaurant in Tokyo
- ordering breakfast at a café in Paris
- checking into a hotel in Sevilla
- ordering bubble tea in Shanghai
- shopping at a convenience store in Seoul late at night
Babbli currently includes six destinations across six languages.
Each scenario has a concrete objective.
For example, in the New York department store scenario, the learner has to find a navy wool scarf for their sister, describe what they are looking for, handle the item being sold out, and successfully complete the purchase.
In Seoul, the learner has to find pimple patches, handle a 1+1 deal, pay, cook ramen at the machine, and eventually head back outside into the midnight rain.
The learner does not simply talk to an AI indefinitely. They must successfully navigate the situation and complete the objective.
Before each response, Babbli gives the learner three possible intentions describing what they need to communicate, such as:
- ask where the scarves are
- ask what the server recommends
- explain that an item is sold out
- ask for more time
The learner chooses what they want to communicate, but Babbli does not give them the sentence.
They must formulate the response themselves in the target language.
Learners can choose between Voice Mode and Text Mode before entering a scenario. Voice Mode focuses on actually speaking through the conversation using push-to-talk or a live conversation, while Text Mode allows learners to practice comprehension, vocabulary, and sentence construction without requiring microphone input. Mandarin Text Mode also supports pinyin, while Korean supports romanization.
The AI character listens to or reads the learner's response, understands what they were trying to communicate, and responds naturally. The conversation can branch depending on what happens: a topping might be sold out, the NPC might misunderstand the learner, a reservation might have a problem, a card might require a second attempt, or an unexpected follow-up question might appear.
Babbli also includes a five-step hint ladder, subtitles, slower speech, native pronunciation references, ambient audio, different difficulty levels, and a customizable Traveler Profile with preferences such as pronouns, avatar settings, favourite destination, and coach voice.
Completing scenarios also earns cosmetic passport pins, with 10 pins to collect.
At the end of a completed scenario, learners receive feedback on comprehension, speaking clarity, fluency, vocabulary, and independence. Voice sessions also provide additional speech analytics such as filler-word usage, pauses, speaking rate, and response timing so learners can better understand how naturally and confidently they communicated.
How we built it
Babbli is built with Next.js, React, TypeScript, Tailwind CSS, Framer Motion, Zustand, and ElevenLabs.
ElevenLabs is deeply integrated throughout the experience.
ElevenLabs ElevenAgents powers Babbli's AI characters. Each character can listen to the learner in real time, understand typed responses in Text Mode, speak naturally in the target language, remember the conversation, ask contextual follow-up questions, and react to the learner's response.
A deterministic scenario state machine sits underneath the AI.
The AI controls how the character speaks, while Babbli controls what must happen for the scenario to progress.
After every learner response, the character reports what it thinks the learner meant back to Babbli's scenario engine through a client tool. The engine then decides whether the learner was understood, whether the scenario should progress, what is available in the environment, and which branch should occur. It then tells the character what to say next, allowing the AI to deliver the response naturally. This prevents the experience from becoming an open-ended chatbot while still allowing conversations to feel dynamic.
We also use ElevenLabs ElevenCreative to create 26 native-sounding voices across the project, including the six main characters, background voices, system voices, and native coaches. Text to Speech powers Repeat and Slow, audio hints, native-speaker reference audio, and character voices in Text Mode, including word-by-word karaoke subtitles. ElevenLabs Sound Effects provide the environmental ambience and individual sound effects throughout the scenes, while Eleven Music is used for the menu theme and the late-night radio playing in the Seoul convenience store.
ElevenLabs Scribe transcribes learner speech and provides word-level timing information. We use this data to evaluate pauses, hesitation, speech rate, response timing, fluency, recognition clarity, and filler words such as "uh," "um," and "erm." It also powers live captions while the learner speaks.
Challenges we ran into
One of the hardest problems was balancing AI freedom with deterministic gameplay.
If the AI controlled the entire conversation, it could easily reveal answers, skip objectives, invent information, or drift away from the scenario.
We solved this by separating the system into two layers:
The state machine controls the game. The AI controls the performance.
This allowed us to keep conversations natural while maintaining control over objectives, progression, branching, and evaluation.
Another challenge was making language practice feel immersive without building a full 3D game.
We instead designed Babbli as a layered first-person experience with immersive SVG environments and characters, animated NPC states, ambient audio, subtitles, speaking indicators, and game-like intention choices. The scenes are animated with CSS and a small custom animation loop rather than a game engine or image files.
We also had to account for language learners speaking more slowly and hesitating more frequently than native speakers. This affected how we designed turn-taking, microphone interactions, speech analysis, and the option to practice through either Voice Mode or Text Mode.
Another challenge was making our evaluation meaningful rather than showing arbitrary percentages. We separated comprehension from speaking performance and used transcript and timing data to explain things such as pauses, speaking rate, filler words, response timing, and assistance usage.
Accomplishments that we're proud of
We are especially proud that Babbli is not simply another AI language chatbot.
Every conversation has a goal, every learner decision can affect what happens next, and the learner must actually formulate the language themselves to progress.
We were also able to build the scenario system so that completely different environments and languages can run on the same underlying engine.
The same architecture can handle buying a gift in a New York department store, ordering ramen in Japanese, navigating a café interaction in French, resolving a hotel reservation problem in Spanish, ordering bubble tea in Mandarin, or navigating a late-night convenience store in Korean without rebuilding the core game for every scenario.
We are also proud of how deeply ElevenLabs became part of the experience: ElevenAgents powers the characters learners interact with, Voice Design creates the voices, Sound Effects and Eleven Music build the environments, and Scribe helps analyze the learner's speech.
What we learned
Building Babbli taught us that realistic language practice is about much more than generating correct sentences.
Timing, uncertainty, listening comprehension, environmental noise, follow-up questions, misunderstandings, hesitation, filler words, and recovery are all part of real communication.
We also learned how conversational AI can be combined with deterministic application logic to create experiences that are both dynamic and measurable.
Separating the AI's conversational performance from Babbli's game state allowed us to keep conversations natural without losing control over objectives, progression, and evaluation.
We also learned that giving learners too much assistance can defeat the purpose of the simulation. This led us to build a progressive hint system where learners can choose how much help they need, while still tracking how much assistance they used.
What's next for Babbli
We want Babbli to eventually become a library of real-world situations that learners can practice before traveling, studying abroad, interviewing, working, or simply speaking with someone in another language.
Future versions could include more locations, regional accents, automatically generated scenarios, deeper persistent progression, vocabulary history, spaced repetition, more detailed pronunciation analysis, expanded avatar customization, achievements, and user-created simulations.
We also want learners to be able to track their performance across multiple conversations and see how their comprehension, fluency, confidence, vocabulary, and speaking habits improve over time.
Our goal is simple:
Practice the conversation before it becomes a real situation.
Built With
- ai
- elevenagents
- elevencreative
- elevenlabs
- framer-motion
- next.js
- react
- scribe
- speech-to-text
- tailwind-css
- typescript
- zustand
Log in or sign up for Devpost to join the conversation.