Sema: Audio description at your pace

Sema helps small film publishers prepare audio description and gives blind and low-vision viewers control over the visual story on Fire TV.

Comment

Inspiration

A film tells its story through more than dialogue. A gesture can change the meaning of a conversation, and a silent discovery can explain what happens next. For someone who cannot see those details, audio description can make them part of the story.

Worldwide, an estimated 43.3 million people were blind and another 295 million had moderate or severe vision impairment in 2020, according to the Global Burden of Disease study.

For a small producer already stretching a budget to finish a film, preparing audio description can be difficult to take on. Someone has to write the descriptions, record them and check every line against the picture and soundtrack. At Cinepros’s published rates of $7–16 per video minute, describing a catalog of ten ten-minute films would work out to $700–1,600, depending on the service and voice options.

We built Sema to help producers prepare that material and give blind and low-vision viewers more control over how they experience it. A publisher gets editable descriptions and measured recordings; a viewer gets to choose how much detail they hear and revisit a visual moment from the Fire TV remote.

What it does

Sema is a Fire TV app running on Vega OS. Its demo includes two free-to-watch, openly licensed 72-second excerpts from Blender’s Sintel and Tears of Steel.

Viewers can choose Essential, Standard or Rich description. In the same Sintel alley, Essential says “A woman walks,” while Rich says “A red-haired woman carries a staff through a narrow sunlit alley.”

When a scene has more to describe than the soundtrack has room for, Revisit pauses the film and plays a description before the viewer continues from the same frame. Earlier moment steps back through nearby moments, while Browse moments offers time ranges and named moments for more distant lookbacks. Only moments whose authored reveal time has passed are offered.

If an important description is interrupted, Sema keeps it available until the viewer hears it or deliberately skips it. Recorded guidance supports remote navigation, and the native host hands announcements to VoiceView when it is enabled.

How we built it

An authoring pipeline prepares the descriptions before release. Amazon S3 holds the authorized source film, and Amazon Transcribe supplies dialogue timestamps. Bedrock Nova Pro observes timestamped frames, while Nova 2 Lite drafts the description levels and separate Revisit descriptions. Amazon Polly generates the recordings.

The pipeline measures each recording against its dialogue gap, requests shorter wording when necessary and checks that required facts survive the rewrite. Production approval binds the source, wording, timing and audio together so that later changes reopen review. The publisher’s editor remains responsible for approving the descriptions.

The player uses JavaScript inside a Vega WebView, with a React Native host for native controls and VoiceView integration. Film, narration and guidance play from local files, so watching and revisiting need no model call or AWS connection.

Our two-excerpt AWS pilot used an estimated $0.32–0.38 per 72-second excerpt, including reruns. Applying those rates to 100 minutes suggests $27–32 in AWS usage. That covers cloud preparation; editing, integration and distribution require a separate budget. It is a different scope from a provider’s finished description service.

Challenges we faced

Getting a description to fit was only part of the problem. Nova sometimes identified the wrong object, and shortening a sentence could remove an important fact. We added required-fact checks and recorded editorial changes against source frames, keeping accuracy decisions separate from duration measurement.

Playback also needed to cope with interruptions. A cancelled recording must never count as heard, and a late-starting clip must not run into dialogue. The scheduler checks fit again when audio starts and preserves interrupted critical facts for recovery.

On the Vega Virtual Device, H.264 and VP9 playback stalled, while VP8/Opus played through. Local fetch/XHR and remote Back handling also differed from the browser, so we bundled package data into a script and routed Back through the native host. Our 26-entry friction log documents the problems, reproductions and workarounds.

What we learned

More detailed narration helps only when there is somewhere to put it. In the Tears of Steel bridge conversation, the longest dialogue gap is 1.21 seconds, while our setting description takes 3.3 seconds. Even a carefully written timed track has to leave some details out. Revisit gives the viewer a way to hear them when they choose.

We also learned to distinguish successful speech routing from an audible screen-reader test. The simulator shows VoiceView state reaching the host and Sema’s own guidance switching off, but its missing TTS engine prevents verification of the resulting VoiceView speech.

Accomplishments we’re proud of

The demo shows the complete viewing loop on the Vega Virtual Device: choosing detail, revisiting the bridge scene, hearing an interrupted description again, browsing earlier moments and continuing from the paused frame. Both excerpts play without cloud credentials or a network connection.

The project has 159 passing JavaScript tests, with 78 offline pipeline tests and six native host tests in the latest respective runs. Publishers can inspect the generated files and use the integration guide to connect the scheduler to another player.

What’s next

Our first intended customer is an independent publisher with rights to its films, an editor and a compatible Fire TV app. We want to take a complete short film through that workflow, measure the editorial effort and integrate the resulting package into a publisher’s player. A larger openly licensed catalog and a Fire OS port would follow.

Film credits

Sintel (2010): © copyright Blender Foundation | durian.blender.org. Tears of Steel (2012): Blender Foundation | mango.blender.org. Film excerpts and adaptations are used under CC BY 3.0, with sources and modifications recorded in the repository. The Blender Foundation does not endorse Sema. Code is MIT-licensed.

Built With

Share this project:

Updates

Submission history