aloud

What it does

Most of us watch television without looking at it for half the film. Some people never see the screen at all. aloud is a Fire TV app, built on Vega OS, that keeps both in the film. The shelf reads each title aloud as the remote moves across it. Press play and, in the silences between the lines, a voice tells you what is on screen: "The flying ship continues descending toward the residential buildings below." Each description is measured into a silence (on Tears of Steel, under a second of overlap with dialogue in 12 minutes, 0.98 s, none on a captioned line), and the film dips a little under the voice so every word is clear.

Missed a description, or watching with narration off beside someone who does not need it? Press Menu: the film pauses, aloud says the most recent description again, and the film carries on. (It repeats the track; it does not look at the picture, so in a long stretch without a description it can be a minute old.) On the lighter "only what changed most" setting, two presses in two minutes tell aloud it is leaving out too much, and it goes back to saying everything, and its help line says why. Narration on or off, voice level, how much is said, play/pause, skips and Back are all on the remote, and every setting says its name and value aloud when the remote reaches it and when select changes it. (The player also requests Vega media-control focus for Alexa transport; that is untested on a device, so it is not shown.) One voice speaks the shelf, the settings and the film, so a viewer who cannot see the screen gets one continuous experience rather than a menu and then a film. For everyone else on the sofa, "Picture: Full screen" turns the player into an ordinary television player with the description as a caption.

Who it is for

Built first for blind and low-vision viewers, for whom it is the difference between following a film and not; useful to everyone else whenever they look away. Audio description exists for a small slice of television because a person writes it by hand, title by title. The US rule (47 CFR §79.3) requires 87.5 hours a quarter from the big four networks' stations in the largest markets and the top five cable networks, about 4% of a channel's airtime; everything else, including most streaming, is voluntary. It matters most to blind and partially sighted viewers, and to a much larger audience watching without looking: the cook, the parent half-watching, the older viewer whose eyesight is going, the viewer following a film in a second language. aloud is aimed at the titles nobody will pay a describer for: FAST channels, archives, independent film, and the long tail of every catalogue.

How it is different from Prime Video's AI

Prime Video uses AI to find the gaps in the dialogue so its production teams can write description scripts faster, for its own catalogue. A person still writes every line. aloud writes the description itself, places it, measures it and plays it on the device. The track is made ahead of time, not live, as a standard WebVTT/JSON file, so a person can still review it before it airs, and the television does no inference at all.

What sets it apart

• Native Vega. React Native for Vega, w3cmedia's Video for the film and a second AudioPlayer classified as accessibility speech for the voice; not a web page in a WebView.
• A reproducible safety number. Overlap with dialogue is measured against the film's own production stems, not the detector that placed the cues: 0.9 s in 12 minutes of Tears of Steel, reproduced independently from fresh downloads.
• Any model, cents a title. The describer is one environment variable. Sintel was described by an open model on a CPU with no key; Tears of Steel by Claude Haiku 4.5 for $0.14, after an open 7B model on a laptop was tried and not shipped.
• The whole app speaks. Titles, settings and every change you make, in the description voice; VoiceView, if on, takes over.
• Reviewable, and gated. Standard WebVTT/JSON. A cue must sit in its gap, finish its sentence, and be about the picture rather than the camera, or it is not spoken and not shipped. Characters are named only from each title's cast list, as the film itself names them, and every sentence is checked against the picture before it ships: 13 of Tears of Steel's 58 were refused in review and their gaps left silent.
• The player is a library. vega-described-track (Open Source mini, below) packages the scheduler and voice for any React Native for Vega player; a patch for Amazon's multi-TV sample is included, not yet run on a device.

How it is different from other AI audio description

3Play Media, Verbit and Audible Sight sell AI-written description to organisations, with human review or editing; ViddyScribe writes it into uploaded videos and narrates YouTube in the browser; YouDescribe gives volunteers AI drafts. aloud's addition is the television half: a native Vega player a blind viewer can run alone, a scheduler that refuses any description the remaining silence cannot hold, and an overlap figure measured against the film's production stems. We found no other tool publishing such a figure.

How it works

1. Listen for speech, three ways. A hand-built speech-band and syllable-rhythm detector, the Silero neural voice detector, and the title's own caption file. A gap must be clear of all three. On Tears of Steel, measured against a dialogue map built from the film's production stems, the hand-built detector alone left 27.4 seconds of dialogue inside its gaps; all three together leave 0.9.
2. Describe what changed. For each gap where the picture has changed, a vision-language model is shown two frames of the shot and the last three things said, and asked for one present-tense sentence of about the number of words the gap can hold. Characters are named from the title's cast list, as the film names them, and nobody else is named. On-screen text is read out. Long action sequences are split at hard cuts. Black frames, sentences that stop before their noun, and sentences about the film rather than in it (the camera, close-ups, logos, credits) are refused, in the pipeline and again when the track is published into the app.
3. Measure, then place. Each description is synthesised and measured, never estimated, and placed inside its gap. The track refuses to be written if any cue would leave its gap.
4. Play it on Fire TV. The Vega app plays the film through w3cmedia's Video, fires each pre-synthesised cue through an AudioPlayer classified as accessibility speech, and decides from currentTime alone whether the rest of the gap can still hold it.

Tears of Steel, the film the video is built on, was described by Claude Haiku 4.5 with its cast list (Thom, Celia, Barley, the Captain): 45 descriptions, 2.2 minutes of narration, of which 0.98 seconds overlap dialogue, measured against the production stems, in six touches of under 0.4 s and none on a captioned line. Sintel (41) and the family comedy Big Buck Bunny (47, no dialogue at all) were described by Qwen3-VL 4B, an open model, on a CPU through llama.cpp with no key and no GPU. Both go through the same OpenAI-compatible describer; Gemini plugs in the same way. Cues that described the photography, read the crew list, or were simply wrong were refused, by the gate or by a reviewer against the frame, and their gaps left silent.

Challenges

• Silence is not the absence of sound. A film's quiet parts are full of score and rain; its loud parts are sometimes where the lines are. The detector that never failed on recorded voices over rain and music failed 30 times on a real film, which is why it now has two more ears.
• Vega gives an app the time, four times a second, and nothing else. No frames, no frame callbacks. So vision runs ahead of time and the player's scheduler refuses to start a description the rest of the gap can no longer hold.
• Models write about the film instead of in it. "Screen is completely black." "Text appears on screen." Both now leave the gap silent.

What's next

A describer review screen for human sign-off; tracks served per title from a catalogue service rather than bundled; describing a household's own library; voices and languages per viewer.

Credits and licences

Tears of Steel, (CC) Blender Foundation, mango.blender.org. Sintel and its trailer, (c) copyright Blender Foundation, durian.blender.org. Big Buck Bunny, (c) copyright 2008, Blender Foundation, www.bigbuckbunny.org. All CC BY 3.0, from download.blender.org, re-encoded; descriptions are a separate track.

Built With

• claude-haiku
• ffmpeg
• jest
• kokoro-tts
• llama.cpp
• numpy
• ollama
• python
• qwen3-vl
• react-native-for-vega
• react-native-kepler
• react-native-w3cmedia
• silero-vad
• typescript
• vega-os