Visible Voices

Nobody misses the film: SDH captions with agents and proof.

Comment

Inspiration

The major platforms have already understood something: a film is not complete if part of the audience cannot follow it. Netflix, Amazon and Disney include subtitles for the deaf and hard of hearing (SDH) in their catalogs and publish style guides for producing them. It is a trend moving in the right direction, and it should reach far beyond the big releases: independent film, festival shorts, regional documentaries. That is where most people are still missing out on the complete film today.

SDH asks for more than transcribing dialogue. You have to identify who is speaking even before they have a name, describe the sounds that matter to the story, mark music and silence, respect reading limits and keep the same criteria from beginning to end, without giving away anything about the plot. It is specialized editorial work, and when the material changes or gets translated, much of that judgment has to be applied all over again.

With Visible Voices we set out to improve that workflow by following industry best practices and, above all, by thinking about the audience these subtitles are for. We placed agents in the stages that take the most time from a post-production team: listening, verifying, keeping everything consistent and translating. And we did it with two fixed rules: every proposal comes with its evidence, and none is approved without a person.

What it does

Visible Voices is a post-production workspace for accessible subtitles. It takes an MP4 with no subtitles, no script and no synopsis, and carries the editor from analysis to export with agents that propose, verify and translate, while approval always stays in human hands. The resulting track follows the SDH format platforms already use: dialogue, speaker labels, sounds, music and vocal delivery, within the reading limits of the style guides.

It understands the film before subtitling it. A coordinator agent analyzes the whole file and proposes a map of scenes, characters, tone, relevant events and the moment each fact is revealed to the viewer.

It separates dialogue from sound because they are different problems. Dialogue comes from Speech-to-Text V2 with word-level timing. Sounds, music and vocalizations come from Gemini listening to excerpts of the original audio. The speech transcript is never used as a sound-effects catalog.

Every caption carries its evidence. For each sound cue, the editor shows the audio excerpt and the frame that support it, and distinguishes acoustic observation from model inference from human approval. Nothing is approved on its own.

It remembers the film to stay consistent. Agents work in chunks, but the film is one piece. So the project keeps a film memory: scenes, characters, proposed relationships, sound motifs and narrative facts, each with its evidence and the moment from which the viewer can know it. When an agent writes the caption at minute eight, it receives what the film has shown up to that point, so the same character gets the same label, the same recurring sound is described with the same criteria, and the editor's decisions about names, phrases and descriptions carry into every following chunk. The editor can correct or retire any memory and see which captions depended on it.

It does not give away the plot. That same memory knows when each fact is revealed. If a voice is heard at second 20 and its name is revealed at second 90, a neutral label is kept before second 90. Agents writing a segment receive only what is knowable at that instant; the full film summary never reaches them.

It reviews accessibility and corrects with verification. An accessibility review goes through the track and produces findings. Each proposed correction is carried out by an agent that listens to the interval again and is validated by a second agent before being saved, always as a draft and with a change journal.

It translates from the approved track, with its own memory. A contextual translator receives the approved English caption together with the project glossary, the film memory and the English-Spanish pairs the editor already approved in earlier scenes, retrieved by semantic similarity. An expression translated one way in the first scene stays that way in the last, names and terms do not drift between chunks, and sound cues are translated with the same criteria. A bilingual reviewer then compares both tracks looking for omissions, additions and reading problems. There are ten language profiles, each with its own reading-speed limits. Every translation is tied to the English revision it came from: if the English is edited or a memory is retired, dependent translations are marked stale until someone reviews them.

It exports and presents. SRT and WebVTT with review status (draft, reviewed, stale), a preview with adjustable size, background and position, and an optional, restrained expressive mode that emphasizes words with confirmed vocal intensity without changing text or timing.

Everything stays visible. The Agent activity section keeps prompts, responses, media sent and timing for every call, including failed attempts.

How we built it

Eleven agents with output contracts. On Google ADK we defined a base agent whose result is validated against a schema before the pipeline continues: FilmCoordinator, AccessibilitySoundObserver, LocalAudioReviewer, LocalVisualReviewer, LocalCaptionReviewer, AccessibilityExperienceReviewer, AccessibilityCorrectionAgent, AccessibilityCorrectionVerifier, ContextualTranslator, BilingualReviewer and TranslationMemoryCurator. Each one receives only the information for its stage; the numeric format validators are deterministic functions, not models.

Google Cloud at runtime. Gemini 3.8 Flash for multimodal understanding, composition, review and translation; Speech-to-Text V2 with the Chirp 3 model for dialogue with word-level timestamps; gemini-embedding-001 for semantic memory; private Cloud Storage for media. There is no other AI provider and no silent fallback model.

Chunking with context. Global analysis first, then 20 to 40 second chunks aligned to pauses and scene changes, with overlapping margins that are reconciled so captions are not duplicated. A sentence that crosses a boundary is analyzed whole.

Blind observation. When verifying a sound, the agent first describes what it hears without seeing the candidate label, then compares. This reduces confirmation bias; it does not eliminate it, and the interface says so.

Memory in PostgreSQL, no separate vector service. Narrative facts, approved phrases, glossary and translation pairs live in tables with their 768-dimension vectors. Retrieval filters first by project, current revision, language, type and temporal permission, and only then computes similarity over that bounded set. The editor sees which memory backed each suggestion and can retire it.

A verifiable editorial profile. The checks apply public rules from the Netflix style guides: 42 characters per line, 20 characters per second for adults and 17 for children's content, two lines, minimum and maximum durations, bracketed labels. This is conformance with an implemented profile, not certification.

Durable work. A PostgreSQL queue with a single owner per session, per-stage checkpoints, idempotent retries and recovery after restarts. Each media version is processed once; a local correction re-analyzes only the affected interval.

Replit end to end. The project was built with Replit Agent and is deployed on a Replit Reserved VM: FastAPI serves the API, the React frontend and the persistent worker through a single port, with Replit PostgreSQL and server-side secrets. FFmpeg prepares the media; Google credentials never reach the client.

Challenges we ran into

STT is not a sound detector. At first we tried to derive sound events from pauses in the transcript. It does not work: a pause cannot tell a door from footsteps. We ended up with two separate analysts and a verifier that cross-checks audio, image, wording and timing.

Seeing something move does not prove it makes a sound. An off-screen sound can be valid and a visible action can be silent. The verifier had to learn not to reject the first or invent audio for the second.

Spoilers get in where you least expect them. Filtering by the caption's timestamp is not enough: a memory approved today may describe a fact revealed at the end. Filters apply to the content of each memory and its dependencies, and the global summary is never handed to the agents that write captions.

A lost negation. In a real test, the transcript dropped a "no" that changed the meaning of a sentence. The correction agent listened to the excerpt again, restored it and the verifier confirmed it; the result was saved as a draft. That case defined the rule: when the composer and the transcript disagree on a negation, the system keeps the draft and raises a warning instead of resolving it silently.

Invalidating the right things. Editing an English caption, retiring a memory or replacing the file must make stale exactly what depends on it and nothing else. Modeling those dependencies per revision was more work than generating the captions.

Accomplishments that we're proud of

  • A complete, reproducible workflow: upload a new video, generate, review with evidence, correct with verification, translate and export, with no simulated results at any stage.
  • Spoiler prevention verified with deterministic temporal-isolation tests on memory retrieval.
  • Twelve backend suites and four frontend suites passing, including dialogue reconciliation, queue lifecycle, concurrent edits and verifier refusal.
  • Complete traces of every agent available to the editor, with failures preserved even when the retry succeeded.

What we learned

A second look by the same model is not independent verification; it helps, but human listening remains the reference. The confidence a model declares is not measured accuracy and should not be presented as such. And accessibility is not solved with more animation: the expressive mode is an experimental preference, not a badge of greater accessibility, until real SDH users evaluate it.

What's next for Visible Voices

  • Evaluation with deaf and hard-of-hearing viewers: comprehension, reading effort and preference between standard and expressive presentation.
  • Measuring omissions, false sounds and timing errors against independent human annotations on more clips.
  • Feature-length films and multiple videos per project; today the limit is three minutes and 100 MiB.
  • Review of the ten language profiles with native speakers.
  • Complete transitive invalidation in translation memory, and scaling with distributed leases and per-user quotas.

Built With

Share this project:

Updates

Submission history