TaleTail

Inspiration

The Cartesia track asked for something where voice is the product. Reading aloud was the clearest case we could think of. Kids learn to read by reading out loud to someone who listens: someone who waits when they get stuck, helps with the hard word, and asks "why do you think that happened?" Reading researchers call this dialogic reading. It works, but it needs a patient adult, one-on-one, every night. A tired parent at 8pm asks "did you like it?", gets "yes", and the moment is gone.

What it does

TaleTail is a talking picture book. The child plays the youngest of the Three Little Pigs.

Say the title to open the book. No tapping. The child says "Three Little Pigs" and the story starts.
A cast of voices. A narrator, two older pigs and the wolf each speak in their own Cartesia voice.
The child reads their own lines. When it's their turn, a line appears in a speech bubble and each word lights up as they say it correctly.
Stuck-word coaching. If the child is stuck on a word for 5 seconds, the narrator says it gently and waits for the child to say it back before moving on.
The pigs ask real questions. Between scenes they ask open questions ("What would you build with?" "Why did it stay standing?") and reply to what the child actually said.
"Hey Foxy." At any point the child can call Foxy and ask a question about the story. Foxy answers using only what has happened so far, so nothing gets spoiled. The child says "I'm done, Foxy" and the story picks up where it left off.

Nothing is stored. Audio is thrown away after it's transcribed, what the child said is wiped when the story ends, and there are no accounts.

How we built it

Cartesia does all the voice work:

Ink 2 (streaming speech-to-text) listens to the child. Its semantic turn detection mattered a lot: kids pause in the middle of a sentence all the time, and a silence timer would cut them off exactly when they're trying hardest. Ink's partial transcripts (turn.update) are what light up each word as it's read, and turn.start is how we notice the child talking over a character (barge-in).
Sonic 3.5 (text-to-speech) speaks all five characters over one WebSocket. Each has its own voice ID, speed and volume. Every line gets its own context_id, so when the child interrupts we drop that context and ignore any audio still arriving for it.
Continuations make Foxy feel quick. Foxy starts speaking a filler line ("Hmm, good question!") right away. Claude's answer then streams in sentence by sentence and is added to the same utterance, so the child never sits through a pause.

The rest of the stack:

A Bun server runs the story script and holds the mic and audio session. It also handles the read-aloud checking, the coaching timer and Foxy's state machine.
The read-aloud checker compares each word with what Ink heard. It accepts homophones, contractions ("I will" for "I'll") and dropped endings ("work" for "worked"), because six-year-olds don't speak like a textbook.
The pigs' replies come from templates matched in code first. Only answers no template covers go to Claude, with a short timeout and a scripted fallback, so the story never stalls.
A Next.js frontend built from our Figma designs draws the illustrated pages, speech bubbles and word highlighting. The server runs on Railway and the web app on Vercel.

Challenges we ran into

We rebuilt the whole product halfway through. Our first design had no screen at all: the child held a paper book, and two characters (Pip and Owl) talked with them through a laptop speaker. Then we realized that if the goal is reading practice, the child has to see the words and the system has to know which word they're on. So we changed course mid-hackathon. We moved to a fixed script, an on-screen read-along, and four story voices with word-by-word highlighting.
"Done speaking" means different things on the server and in the browser. Sonic sends a whole line in about 200ms, but the browser takes several seconds to play it. So the server thought a character had finished while the child could still hear them, and barge-in broke. We fixed it by having the server always send a stop signal and letting the browser decide whether anything is actually playing.
Foxy kept waking itself up. The narrator says "say hey Foxy if you want to ask a question", and Ink heard it through the speakers. We had to tell the child's "hey Foxy" apart from the story's own, and handle the race conditions of a question arriving while a line is playing, mid-read, or right as Foxy is dismissed.
Kids don't wait their turn. They talk over characters, trail off, and repeat words. Every step of the script had to cope with a turn starting at the wrong moment without skipping the story or getting stuck.
Speaker echo. Without headphones, the characters' voices leaked back into the mic and were taken as the child's speech.

Accomplishments that we're proud of

Foxy. A child can interrupt the story at any moment with a question and get an answer that knows the story so far but won't spoil the ending. Thanks to Sonic continuations, Foxy starts talking right away instead of leaving a gap while Claude thinks. It also handles the messy cases: a question in the middle of a line, during a read-aloud, or "I'm finished" instead of "I'm done".
Stuck-word coaching. This is the part that feels most like a real reading buddy. The story doesn't read the line for the child or skip ahead. It waits, says the one word they're stuck on, and keeps going only once the child has said it themselves.
A story that never goes silent. If a voice fails, the line is retried in the narrator's voice. If Claude is slow, a scripted reply takes over. Nothing a child does should leave them staring at a frozen screen.

What we learned

With kids, timing is the whole interface. When to wait, when to help, when to let them interrupt and when to keep going: those choices mattered more than any model. Ink's turn detection did more for the experience than any prompt we wrote.
Templates beat a model for predictable moments. Most kids' answers fall into a few groups (a material, a reason, "I don't know"). Answering those from code is instant and safe, which leaves the model for the answers that are actually surprising.
Streaming hides latency. Continuations let us start speaking before we knew the whole answer, and that changed how responsive Foxy felt.
Voice products for kids raise serious privacy questions. We designed for "nothing stored on our side" from the start. We also learned that true zero data retention with our vendors would be the first thing to sort out for a real product.

What's next for TaleTail

More public-domain stories, then a general mode that can read along with any book.
Difficulty that adapts to the child's age and reading level: longer lines, harder words, and less coaching as they improve.
A summary for parents: not a score, just which words their child struggled with and what they said about the story.
Testing with real kids and early-childhood educators. We built this on dialogic reading research, but we haven't tested it with children yet. That's the step that turns a good design into a proven one.