Polytale

Conversation
Language Selection
Phrase helper
Story ending

Inspiration

The World Cup brings together people from different countries who share a love of football but may not share a language. Meeting other fans, asking for directions, or joining a conversation can mean trying to communicate with only a few words.

We built Polytale to give people a place to practice those moments. Players enter a story, meet characters, and use another language to get something done. There is room to make mistakes, ask for help, and keep trying.

Our goal is to make speaking feel approachable even before a learner feels confident. A scene gives their words context, and a character’s response gives them a reason to continue.

What it does

Polytale is a language-learning adventure where players speak or type to AI characters and move the story forward through conversation.

In our current adventure, Match Night, your friend Mei has your ticket, your phone is dead, and your only lead is a crowded bar on World Cup final night. The fans know her, but they speak the language you are learning. You have to introduce yourself, ask about your friend, and join the conversation to discover where she went.

Players can choose Spanish, Mandarin Chinese, Japanese, or Korean. Characters respond with spoken audio and target-language subtitles, with pronunciation guides where appropriate.

Help is available without leaving the scene. Players can click to reveal a translation beneath a character’s line, ask for a slower repeat, or request a hint. If they know what they want to say but cannot find the words, “How do I say…?” provides a phrase they can listen to and try themselves.

Conversations reveal clues and complete story objectives. At the end, a vocabulary recap shows the words players encountered and whether they used them independently or needed help.

How we built it

We built the frontend with React, TypeScript, and Tailwind CSS, with a Python and FastAPI backend.

Google Gemini API drives the roleplay. Each turn includes the character’s personality, the scene, conversation history, and current game state. Gemini responds to the player and uses function calls to request changes, such as revealing a clue or updating an objective. Our Python executors validate those requests before applying them.

Gemini also powers the phrase helper, returning expressions with structured pronunciation and word-level information for the interface.

ElevenLabs handles both listening and speaking. We use ElevenLabs text-to-speech for character dialogue and Scribe speech-to-text for player responses. A transcription preview lets players check what the system heard before their response enters the conversation. They can also replay spoken phrases before trying them aloud.

Language content lives in separate JSON files, allowing the same engine and scene to support four languages. We prepare the artwork ahead of time and generate dialogue during play. The server records exchanges and assistance use to produce the vocabulary recap.

Challenges we ran into

One challenge was deciding how much support to provide. We wanted players to try interpreting unfamiliar language, but getting completely stuck should not end the experience. We made translations optional and separated help with understanding the character from help with forming a response.

Keeping the roleplay consistent was another challenge. Players can say anything, including things that do not follow the expected plot. The agent needs to respond naturally while respecting the conditions for revealing clues and advancing the story.

We addressed this with an agent harness that separates dialogue from validated state changes. We also tested it adversarially, using inputs that went off the expected story path to check whether the agent could bypass those conditions. In the cases we tested, the harness kept story progression within the game’s rules while allowing the character to respond.

Voice interaction required coordinating recording, transcription, model responses, and audio playback. We built distinct interface states and a transcription preview so players could understand what was happening and control what they sent.

Accomplishments that we're proud of

We built a playable adventure that combines spoken interaction, story progression, and vocabulary feedback across four languages.

Gemini and ElevenLabs are central to that experience. Gemini lets the character react to what a player actually says, while ElevenLabs lets players practice listening and responding aloud within the same exchange.

We are also proud of the structure around the roleplay. Validated tools and adversarial testing helped us keep the story consistent when players moved away from the expected conversation.

Most of all, we made room for learners to participate with the language they have. A short phrase, an imperfect response, or a request for help can still be a step forward in the story.