SpellSpoken

Inspiration

Every language app is good at helping you recognize a word, tap the right tile, keep a streak. The part that actually makes you fluent. i.e. opening your mouth and being judged on the sound, is the part they skip because its awkward and nobody likes being told they are wrong. We wanted a game where your pronunciation is the subject being sharpened.

What it does

SpellSpoken is a deck-building roguelike where every card is a spell and every spell is a phrase in the language you are learning. Pick a card, the game draws an incantation (a word, a phrase or a sentence, by card tier), say it into the mic, then a pronunciation scorer grades it.

Chain one element for bonus damage or pair elements into reactions, play keyword cards against enemies, collect relics, and climb a branching eight-floor map with one life. Words that beat you come back as grudges until you beat them.

Gemini coaches each cast by listening to the actual audio, and it steps in as a second ear when the browser’s recognizer drops a short word. Eco, our instant AI mentor, will answer any question you throw at it. Everything is bilingual, with pinyin on every Chinese string, and the whole game is playable without an API key or a microphone if you just type the incantation.

How we built it

Stack: We used Next.js 15, React 19, and TypeScript, with Tailwind CSS for the candlelit/fairytail look. It’s deployed on Vercel, and the Gemini key lives in a Vercel environment variable. Your runs, mastery, and the lexicon you’ve grown are stored in localStorage, separately for each language.

Hearing you: Casting runs on Chrome’s Web Speech API with interim results and five alternatives. We wrapped it in a small capture state machine that accepts early as soon as an interim result already matches the phrase, stops a beat after you go quiet, never cuts off a sentence mid-pause, and releases the mic on every exit path. At the same time, a MediaRecorder saves an Opus clip of the cast and a WebAudio analyser drives the volume meter.

Scoring: Spanish is compared word by word using accent-insensitive edit distance. Mandarin is converted to pinyin syllables with tone numbers (using pinyin-pro) and compared syllable by syllable, so homophones count and a wrong tone costs a third of a syllable. A sliding window forgives repetition, so saying 塔塔 for 塔 is still perfect. The recognizer’s alternative rank and confidence also weight the result, which keeps an anglicized reading from earning a Perfecto. Numerals get expanded too (“10” becomes diez), since recognizers write numbers as digits. All of this happens deterministically, and the outcome is settled before any model is called.

Gemini does four jobs:

It listens to the clip and gives one coaching sentence per cast.
It plays Eco, who can see the card you’ve selected and its incantation.
Once per floor, it writes and validates 15 new lexicon entries in a theme your deck uses.
When Chrome throws away a lone syllable, it transcribes the clip verbatim, without being told the target, and hands that to the same scorer to judge. The caller has model fallback lists, thinking disabled, timeouts, retries, and a per-model cooldown on 429s. Every failure falls back to template text, so the game never waits on Gemini and Gemini can never change damage.

The game: The engine is a pure module in the style of Slay the Spire. You get 3 breath a turn and a hand of 5, with 40 spells that carry keywords, statuses, element chains, and reactions. There are 13 enemies, 22 relics, 8 events, and a seeded 8-floor map. Because the engine is pure, a bot can play 400 runs at each pronunciation-accuracy level to tune the balance, and 42 assertions guard the rules and the scorer.

Challenges we ran into

Chrome drops a lone syllable. Saying 塔 once produced no usable result, even though the mic meter clearly moved. The team recorded each clip’s peak volume, and when the recognizer returned nothing for a clip with clear voice, Gemini transcribed it without knowing the target, and the normal scorer judged the result.

Our speech recognizer is too kind. Spanish said with English vowels got snapped to the nearest real word, so a bad “gato” looked perfect. Weighting the result by the alternative’s rank and Chrome’s confidence fixed it, and the audio coach now calls out English reading habits.

Our resource economy was too loose, so players could empty their hand almost every turn. With no tension between options, there were no real decisions, and the cost curve was effectively flat. We tightened it by capping the hand at 5, decoupling card cost from tier, and adding keyword cards that compete for the same breath budget. Full-hand turns now happen about one turn in eight, so players have to prioritize.

What’s next for SpellSpoken

Adding more languages to the game. Japanese and French are the obvious next two.

Implementing daily seeded runs. These would come with a shared leaderboard, plus enemy voice lines and a .tech domain.

Finally having a classroom mode. A teacher’s word list would become the lexicon, and mastery would sync across devices instead of living in one browser.