What Did I Miss?

Inspiration

Movie night with friends: someone takes a call or goes to the kitchen, comes back, and is lost. Alone you'd just rewind, but with a room full of people nobody wants to replay five minutes. Asking out loud gets you "shh", or a spoiler.

What it does

What Did I Miss? is a TV app for Fire TV. While a film plays, one button on the remote opens "What you missed":

• A story recap of exactly the part you missed. If you skipped ahead, it covers the skipped range; otherwise the last five minutes. It reads like a story ("Sintel nurses the wounded baby dragon. She calls him Scales. The great dragon carries Scales off while Sintel reaches after him.") rather than a list of shots.
• The one thing that matters, flagged as IMPORTANT ("A huge dragon takes Scales from Sintel. Finding it becomes her whole quest.").
• A visual recap of the last 2, 5 or 10 minutes: real frames from the film with a one-line caption each, stepped through with the D-pad.
• Follow-up questions from the remote, no typing: Why is that important? What clues do we know? Who is involved? What happened earlier?
• Watch Missed Part jumps back to where you drifted off; Continue Watching carries on.

Every answer is spoiler-safe: it only knows the film up to the second you're at. At 9:50 it says nothing about the twist; after the twist plays, it can.

How we built it

• TV app: Kotlin, Media3 ExoPlayer with HLS, a MediaSession for the remote's transport keys, and a remote-first UI (one button, D-pad chips, BACK to dismiss). Fast-forward presses that follow each other merge into one "skip", which is what the recap covers.
• Story index, built once per film: frames every 3 seconds (238 for Sintel), described and checked shot by shot, then condensed into timestamped scenes, events, clues, characters, objects and story states. Each event has a visual description and a plain story line.
• Backend: TypeScript and Express. Every lookup is bounded by the viewer's position (maxTimestampMs = currentPosition), so later events never reach the answer at all. A second layer, SpoilerGuard, checks the finished text against future events and rewrites it if anything leaks.
• Answer providers: a deterministic provider (default; no keys, works offline, fully testable), with Anthropic Claude and Amazon Bedrock providers behind the same interface.
• 30 automated tests, including spoiler checks at fixed points around the twist.

Challenges we ran into

• The first story index was partly invented: it described a forge, a sword and a map that aren't in Sintel. We threw it out and rebuilt every scene and event from the actual frames.
• A real spoiler leak: a scene summary and one clue described the twist before it happened. Fixing it meant building recaps only from time-gated events, never raw scene text.
• The Android TV emulator's H.264 decoder crashes on the 1080p High-profile stream, so the demo uses a 720p Main-profile HLS copy.
• Frame timestamps were 1.5 s early, which put captions on the wrong shots until we measured and corrected it.

Accomplishments that we're proud of

The spoiler boundary holds where it matters most: right before Sintel's twist the app stays quiet, and once the twist plays the recap can explain it. And the recap reads like someone who watched the film, not a caption generator.

What we learned

Spoiler safety has to be designed into retrieval, not bolted on as a filter. You also can't gate a twist you haven't pinned down to the second, so the index has to come from actually watching the film.

What's next for What Did I Miss?

• Testing on a physical Fire TV stick and publishing to the Amazon Appstore.
• Voice: "Alexa, what did I miss?", with the recap read aloud.
• Indexing more titles, and per-viewer recaps for watch parties ("what did I miss since I left?").

Demo film: Sintel, © Blender Foundation (durian.blender.org), CC BY 3.0.

Built With

• amazon-bedrock
• android
• android-tv
• anthropic-claude
• exoplayer
• express.js
• ffmpeg
• fire-tv
• hls
• kotlin
• media3
• mediasession
• node.js
• typescript
• vitest
• zod