Described

Audio description for films that have none. Qwen3-VL on Amazon Bedrock describes each shot, Polly speaks it between the lines, and Fire TV's own audio menu turns it on.

Comment

Inspiration

Audio description a voice telling you what's on screen between the lines of dialogue is how blind and low-vision people watch films. It is also expensive: professional description costs $15–75 per minute (3Play Media). So it goes where the money is. In 2025 ZDF described 99.5 % of its prime-time fiction but only 32 % of its main channel and 12 % of ZDFneo (ZDF). Archive films, shorts and library catalogues never get a describer. Since 28 June 2025 the European Accessibility Act also covers the apps people watch through. We wanted to see whether a cloud pipeline could draft description for that long tail at a few cents a minute, and deliver it in the place viewers already look: the TV's own audio menu.

What it does

Described is a Fire TV app. Pick a film, press play, and in the gaps between dialogue a calm voice tells you who is there, what they do, where it happens, and any words that appear on screen.

  • Description is on by default. Everything works with the D-pad, Select, Back and Menu, and every control has a VoiceView label.
  • Fire TV's own selector is the switch. The narration is a second audio track in the stream, marked as description (public.accessibility.describes-video). Menu shows Audio — Original · Audio description and Captions — Off · Captions · Rich captions · Description text.
  • Extended mode pauses the film when there's more to say than the gap allows, speaks, then carries on — the behaviour WCAG 1.2.7 describes. You can turn it off.
  • Rich captions add evidenced sounds in brackets ([creature roars]) for viewers who are deaf or hard of hearing.
  • Captions follow your Fire TV caption settings, clamped to WCAG AA contrast.

Six Creative Commons Blender titles are live, including the whole of Sintel (14:48, 117 cues).

How we built it

Pipeline (packages/pipeline, pg-boss jobs behind an Express API):

  1. Shots — ffmpeg scene detection.
  2. Speech map — Amazon Transcribe word timings find the gaps where narration may speak.
  3. Describe — Qwen3-VL 235B on Amazon Bedrock sees 3–6 key frames per shot, with a prompt built on the Netflix, Prime Video and DCMP description guidelines: present tense, objective, no names before they're revealed.
  4. Verify — the same frames and the draft go back to Qwen3-VL, which may only remove what the frames don't show.
  5. Scenes + fit — Amazon Nova Lite groups shots into places with a people-and-places glossary, then shortens each line to fit its gap.
  6. Narrative pass — DeepSeek v3.2 on Bedrock joins a scene's cues into one connected narration. Deterministic guards (facts kept, names only after they're revealed, pronouns only with a clear referent) reject any rewrite that breaks a rule.
  7. Voice + mix — Amazon Polly (generative voice Ruth). ffmpeg ducks the soundtrack 9 dB under the voice and normalises to −24 LUFS.
  8. Package + publish — Shaka Packager writes HLS with two audio tracks and up to three WebVTT tracks. S3 + CloudFront serve it.

App — React Native + Expo on Fire OS, playing through our open-source library @moiz0309/vega-media-kit (player wrapper, WebVTT cue overlay, HLS/VTT parsers). Fire TV integrations: deep links, an Android MediaSession module (remote media keys and Alexa transport), Fire TV caption settings via CaptioningManager, and VoiceView support.

Hosting — the release APK talks to an AWS Lambda function URL backed by Amazon RDS for PostgreSQL, all defined in AWS CDK.

Cost — about $0.09 per minute of film end to end at list prices (≈ $5 per hour). The whole of Sintel cost $1.10.

Challenges we ran into

  • The first model invented things. Our quality gate is ≥ 70 % usable shots, rated blind on faithfulness, timing, style, cleanliness and fit. Nova Pro scored 41 %. We ran a blind bake-off on the same 29 shots: Qwen3-VL 75.9 %, TwelveLabs Pegasus 44.8 %, Nova Pro 17.2 %, Nova 2 Lite 3.4 %. We measured and switched rather than assume.
  • Passing on a clip we tuned on wasn't enough. We added a held-out clip (Tears of Steel) that nothing was tuned on, and the pipeline failed it four times in a row. Fixes followed: a verify pass, legible-text-only rules, a voice-activity check for unlabelled speech, and complete-sentence guards. Then the narrative pass got it through.
  • Narration that talks over people. Getting cues to land in gaps, never over a line of dialogue, took several rounds of timing and overlap checks.
  • Fire TV specifics: ExoPlayer resets its audio decoder on a track switch (we now fade out, hold ~150 ms and fade back in). Alexa voice transport needed a manifest permission that no tutorial mentioned.

Accomplishments that we're proud of

  • The quality gate passed on both clips: 82.8 % on Sintel and 77.4 % on the held-out clip, each rated blind by three independent raters.
  • Description delivered through the platform's own audio-track selector, so viewers don't learn a new control.
  • An open-source media kit published to npm, plus an upstream fix to Amazon's react-native-multi-tv-app-sample (#136, enables ExoPlayer HLS).
  • Roughly 1/200th of the cost of the cheapest professional quote.

What we learned

  • Visual models differ hugely on faithfulness, and only a blind, rubric-based evaluation shows it.
  • Guardrails beat prompts. Every LLM rewrite passes deterministic checks or is thrown away.
  • Description is a timing problem as much as a language problem.
  • Accessibility on TV is physical: focus must change shape and size, not just colour.

What's next

  • Sound-event detection so Rich captions cover more than described sounds.
  • More languages (German voices are already wired).
  • Fire TV catalog integration and search.
  • The Vega OS build (the kit's Vega adapter is experimental today).
  • A review tool so a human describer can approve and edit the draft. Human-authored description stays the gold standard: Described drafts it where none exists.

Built With

Share this project:

Updates

Submission history