EveryWord

Subtitles show what was said, not which word is being said. EveryWord lights each word as it's spoken on Fire TV, turning video kids already watch into reading practice.

Comment

Inspiration

Subtitles show you what someone said. They don't show you which word they're saying right now.

That small gap matters more than it sounds. In India, a simple fix for it has been running on national television since 1996: Same Language Subtitling, subtitles in the same language as the audio, lit up word by word like karaoke, on the films and songs people already watch. It reaches an estimated 200 million viewers [1].

And it works. A five-year study of 13,000 people found that 32 percentage points more children became good readers when they watched with it [1]. It worked because nobody had to sign up: the reading practice rode on programmes people already chose to watch. More than 100 studies also show captions help comprehension, attention and memory, with people learning to read gaining the most [2].

The need is close to home too. In the US, 54% of adults, about 130 million people, read below a sixth-grade level [3], and the number at the lowest literacy level grew from 48 million to 58.9 million between 2017 and 2023 [4].

Amazon already does this for books with Immersion Reading. No TV platform does it for video, because every subtitle file in the world only knows when a line starts, not when each word is spoken.

So we built the missing piece.

What it does

EveryWord takes the subtitles a video already has and adds the time of each word, so the word being spoken can light up as it's spoken.

  1. Lights each word as it's spoken, on Fire TV. Sintel, an open film, plays with the exact subtitles its makers shipped, and each word lights in time with the voice.
  2. Never changes what the subtitle author wrote. Amazon Transcribe only decides when to light a word, never which words appear. Even if it mishears a word, it can't change what a child reads.
  3. Works with the remote. Pick a story from the library on Fire TV and every word lights as it's read. The remote's play, pause, skip and Back keys all work.
  4. Helps a reader keep up. Replay a line, slow it down without changing the voice's pitch, make the text bigger, or switch to a font designed for easier reading.
  5. Counts the words actually read along. Not minutes played: the words the highlight passed. A parent can ask an assistant through Alexa+, "how far did my child get today?", and get the number the TV itself reported.
  6. Explains a tricky word. Through Alexa+, a reader can ask what a word means, explained in the sentence it appeared in by Claude on Amazon Bedrock.

The library today is the film Sintel and five Aesop fables. But nothing ties EveryWord to them: give it any video with subtitles and Amazon Transcribe adds the timing; give it public domain text and Amazon Polly narrates it and times every word. Both make the same word-timed caption format.

How well does it work?

A reading tool has to be right about timing, because lighting the next word too early teaches a child the wrong word-to-sound match. So we measured it on recorded speech we didn't make (LibriSpeech):

  • The highlight lands within 30 milliseconds of the spoken word at the median, about one frame of video.
  • In 766 words, not one lit more than 150 milliseconds early.
  • On Sintel, a film nobody on our team made, 74 of 75 subtitle words were matched to the audio, and the text on screen is identical to the file the film shipped with.

We don't claim EveryWord has taught anyone to read. The 32-point result belongs to Indian television. What's ours is making that idea work on video that already exists.

Try it yourself

On the live site you can:

  • Watch Sintel light up, word by word, with its original subtitles.
  • See the before and after: the plain subtitle file and the word-timed one, side by side.
  • Open a live Alexa+ session to the deployed MCP server and see the reading progress the TV reported.
  • Get your own API key in one click, free and with no account. Paste it into any MCP client, and the readers you name with it are yours alone; the page proves it by asking for the same reader with no key. No key still works for everything.

The Fire TV app is a release APK. The demo video shows it running on Amazon's hosted Fire TV in Appstore Quality Central.

How we built it

  • Fire TV: a React Native TV app for Fire OS with a 10-foot interface, driven entirely by the D-pad. Tested on Amazon's hosted Fire TV, which caught a Back-key bug the emulator never showed. We fixed it in v0.3.2.
  • Alexa+: a self-hosted MCP server (spec 2025-11-25, Streamable HTTP) with six tools: browse the library, recommend a story, record and report reading progress, read a story's text, and explain a word.
  • AWS: Amazon Transcribe for word timings, Amazon Polly for narrated stories, Amazon Bedrock (Claude) for word explanations, plus Lambda, DynamoDB, S3 and CloudFront, all deployed by one CDK stack. Adding timing to a film costs about two cents a minute, once, for every future viewer.
  • Open source: two new MIT packages on npm, so any video app can add word-by-word captions: karaoke-captions-react (the renderer, for web and React Native) and @everyword/captions-core (the caption format and timing engine, with a standard WebVTT export).

Privacy

EveryWord reads to a child; it never listens. There is no microphone anywhere in the web app, the TV app or the server. A reading session keeps five fields and nothing that identifies a child.

Challenges we ran into

  • Our first measurement said 11.5% of words matched. Nothing had crashed: some test audio downloads had arrived cut short while still reporting success. Checking every file fixed it, and the real match rate was 97.2%. Lesson: verify your data before trusting your numbers.
  • Building a Fire TV app on Windows. Long file paths broke the Android build, the usual workaround broke the bundler, and sharing code across the project loaded React twice. Each took a non-obvious fix, all written up for other developers.
  • Vega OS had no path from Windows. The Vega SDK doesn't support Windows, so we built for Fire OS, which runs on today's Fire TV devices.

Accomplishments that we're proud of

  • Word timing measured against real speech, at about one frame of accuracy.
  • Not a single word changed from what the subtitle author wrote.
  • A reading count the TV itself reports, which an assistant can answer questions about.

What we learned

The idea is decades old and the evidence is strong. What was missing was a cheap, automatic way to time every word in a video. That now costs about two cents a minute.

What's next

A bigger library of read-along stories, tapping a word to hear its meaning on the TV, and the case this project exists to make: platforms can turn this on for video today, and Amazon can do it first because it already built it for books.

Sources

  1. Kothari and Bandyopadhyay, 2014, catalogued by the UNESCO Institute for Lifelong Learning, Reading a Billion: Same Language Subtitling, India
  2. Gernsbacher, Video Captions Benefit Everyone, Policy Insights from the Behavioral and Brain Sciences, 2015, PMC5214590
  3. Gallup for the Barbara Bush Foundation for Family Literacy, 2020, study
  4. OECD PIAAC Cycle 2, US results, American Institutes for Research

Every claim with its full source: EVIDENCE.md

Built With

Share this project:

Updates

Submission history