Moment

Pause your Fire TV and ask what you missed. Moment answers from the last seconds of video and subtitles, says "I'm not sure" when it can't tell, and was tuned with a measured experiment.

Comment

Inspiration

You look away for a second and miss something: who just walked in, what a sign said, what actually happened. Rewinding with a TV remote is clumsy, and you often don't know where to rewind to. I wanted to simply ask the TV about the moment I just missed and get a short answer I could trust, including an honest "I'm not sure" when the picture doesn't show it.

What it does

Moment is a Fire TV app (Vega OS). While a video plays, press Menu. The video pauses and five questions appear:

  • What just happened?
  • What did they just say?
  • Who is on screen?
  • What does the text say?
  • What should I notice here?
    Pick one with the D-pad and a short answer appears on screen, with a note about what it was based on (for example "Based onlast 4 seconds of video and recent dialogue (subtiou can ask another question about the same moment, replay the last 10 seconds, or resume. A start screen lets you pick one of six clips. When the question panel opens, thapp **pre-answers the questions in the background*few seconds later is usually already there.

"What did they just say?" quotes the clip's last sd (no AI call) and says honestly when a clip has no subtitles. It's AI-enhanced viewing for when you looked away, are multitasking, or can't read small text. It has screen-realabels and a larger-text option, but I have **not*w-vision, deaf or hard-of-hearing viewers, so Idon't claim it is built for them.

How I built it

  • App: a Vega OS app in React Native 0.83 (TypeScript), playing bundled clips with react-native-w3cmedia, built and ron the Vega Virtual Device, and tested with inso every check is repeatable.
  • Backend: a small Node 24 service (TypeScript). For a pause at time T it selects up to five frames from the last four seconds plus the dialogue from the last ten seconder the pause), builds a prompt, and makes oneAmazon Bedrock Converse call to Amazon Nova Lite per question. Around that call: an answer cache, a timeout with one retry (4 s + 3 s), request IDs, background prefetch, and a ve calls past $130.
  • Clips: six excerpts of Creative Commons or public-domain films (audio removed), prepared with ffmpeg.

The technical core: I measured which context to send

Which frames and subtitle lines to send is a design choice, so I tested it instead of guessing. I wrote 50 questions with expected answers (action, identity, on-screen textle" where "I'm not sure" is correct), kept 35locked as a test set, and compared eight configurations: 1, 3 or 5 frames, with and without subtitles, a wider window, a bigger model, and a revised prompt.

Setup (35 locked test questions) Accuracy Haure" p95 latency Cost per question
Chosen default 73% 3% 91% 3.2 s
Earlier default 61% 3% 82% 4.1 s
Nova Pro, earlier context 69% 3% 82% 5.0

Paired by question, the chosen default beat the eand lost none. A model 13 times the price was notbetter.

A second check: 24 brand-new questions the prompt was never tuned on scored 63% overall (action questions, the weakest category, 54%).

What didn't work, and I report it: I tried thrjust happened?" (a "what changed" prompt, 8 framesover 8 seconds, and sending the last 4 seconds as a video clip). None beat the default and two added timeouts or invented actions, so I kept the default.

Speed: a cold answer takes about 2.4 to 2.9 seive runs). Asked a few seconds after the panelopens, the prefetched answer is usually already there (median 2 to 30 ms), though the slowest cases can be as slow as a cold answer (95th percentile 0.8 to 4.2 s). Prefetch col open. The whole project, including everyexperiment, cost about $0.55 over about 1,400 live Bedrock calls.

Challenges

  • Getting the emulator and build chain reliable on Apple Silicon (a boot hang, and a build that stalled silently on a macOS permission prompt).
  • A player quirk: currentTime read 0 right after pause(), which silently sent wrong timestamps until I read the time first.
  • My first prompt was too cautious and refused queanswered; later, dialogue from after the pauseleaked into answers. Both were found by the evaluation, not by eye.
  • Honest grading: answers were graded by an automacted answers I confirmed. When I re-graded 15 ofthem myself, we gave the same grade on 8 and were within one level on 14; the assistant was somewhat more generous, so treat the accuracy figures as slightly optimistic. The fecords each problem with a workaround and asuggestion.

What I learned

More frames or a bigger model were not the answer. Prompt wording and which dialogue to include mattered more: labelling subtitles as audio (not on-screen text) and never r the pause fixed two failure modes at once. Also,saying "we tried this and it didn't help" is worth as much as the wins.

AWS services used

  • Amazon Bedrock (Converse API): one multi-image call per question to amazon.nova-lite-v1:0 in us-east-1, through the AWS SDK for JavaScript v3. Token usage from each respog.
  • Amazon Nova Pro: used in the experiment as the "bigger model" comparison.
  • AWS IAM: a least-privilege user; credentialsnd are never committed.

No other AWS services are used and nothing is left

Honest limits

Small samples (4 to 11 questions per category). Prps only, not live streaming. Audio is not analysed(the clips are silent), so dialogue comes from subtitles. No user tests with outside viewers yet.

What's next

Real streaming content with on-device frame capture, better answers for action questions, speech transcription for videos without subtitles, and real user tests, including af and hard-of-hearing viewers.

Clips: Tears of Steel, Sintel, Big Buck BunnyLlama Drama (c) Blender Foundation, CC-BY, audioremoved; A Trip Down Market Street (1906), public domain. Full credits are in the repository.

Built with (type each and press Enter): fire-tv, vega-os, vega-sdk, react-native, typescrik, amazon-nova, aws-sdk, aws-iam, ffmpeg, jest,react, hackathon

Built With

Share this project:

Updates

Submission history