Redub

Inspiration

Stories can travel across borders, but traditional dubbing is expensive, slow, and difficult to scale. Many automated dubbing tools translate speech, yet lose the identity of each character, ignore the timing of the original scene, flatten the music and effects, or provide no reliable record of how the final result was created.

We built Redub to treat dubbing as a complete media production workflow. Our goal was simple: dub any series into any language, keep every voice consistent, fit every cut, and prove exactly what shipped.

What it does

Redub transforms an uploaded episode into a broadcast-ready multilingual dub.

It transcribes the dialogue, detects speakers, and creates one stable voice casting table for the entire episode. Every target language uses this same casting table, so each character keeps a consistent identity across languages and retries.

For every dialogue line, Redub calculates the available time from the original timestamps. It estimates a language-specific syllable budget, translates the line, generates the voice, and measures the real audio duration. If the result is too long or too short, an agent loop uses the measured difference to revise the translation and try again:

translate → synthesize → measure → accept / retry / flag

Accepted takes are placed at their exact original timecodes. Small timing differences are corrected with pitch-preserving audio adjustment. Lines that still do not fit are sent for human review instead of being silently included.

Redub preserves the original music and effects track, mixes the new dialogue over it, applies gentle audio ducking, and combines the final audio with the original video.

Every generated attempt produces a Genblaze manifest. The final master and its language provenance manifest are stored in Backblaze B2 with Object Lock. Redub can recompute hashes, verify every shipped take, and show exactly which assets and model runs created the final video.

How we built it

The control plane uses FastAPI with a browser dashboard built in JavaScript, HTML, and CSS. Server-Sent Events provide a live view of transcription, casting, translation, voice generation, retries, review, assembly, and verification.

Genblaze orchestrates the generative media pipeline:

INGEST → SEGMENT & CAST → ⟦per language⟧ → ASSEMBLE → SEAL

AssemblyAI provides speech recognition, word timestamps, and speaker diarization. NVIDIA Nemotron handles duration-aware translation. Redub generates dialogue through Google Cloud Text-to-Speech and Gemini, with a local voice provider as a fallback.

FFmpeg and FFprobe handle audio extraction, duration measurement, timeline assembly, pitch-preserving timing correction, audio ducking, mixing, and final video creation.

Backblaze B2 stores source media, generated takes, manifests, final masters, and provenance records. Content hashes and Object Lock make the shipped output verifiable and resistant to silent replacement.

Challenges we faced

The hardest challenge was timing. A translation can be accurate as text but still take far too long to speak. Character counts and estimated syllables were not enough, so Redub measures every generated audio file and feeds the real timing error back into the next translation attempt.

Maintaining character identity was another major challenge. Voice selection can easily drift between languages or provider retries. We solved this with a single casting table that is created once per episode and referenced by every language branch.

Audio assembly also required careful work. Generated dialogue had to land at exact timestamps without changing pitch or destroying the original atmosphere. We built a timeline assembly process that preserves the music and effects track while making space for the new dialogue.

Finally, tracking provenance across many retries was difficult. We used Genblaze manifests and parent run relationships so the retry history itself becomes the lineage record.

Accomplishments that we are proud of

We built a complete workflow that goes beyond basic translation and voice generation. Redub handles casting, timing evaluation, automatic retries, human review, audio mixing, master creation, and verification in one application.

We are especially proud that every shipped line can be traced back to its exact translation, voice, provider, model, generated asset, and manifest.

Redub also supports graceful provider fallback. If a cloud service is unavailable, the pipeline can move to another provider or use the local implementation. This allows the complete experience to run even without external credentials.

What we learned

We learned that strong generative media systems need measurable evaluations. Listening quality matters, but duration, timeline placement, asset hashes, and provenance can all be verified automatically.

We also learned that provenance works best when it is created during generation rather than added after the project is finished. By recording every attempt as part of the pipeline, verification becomes a natural feature instead of a separate audit process.

Most importantly, we learned that useful automation should keep humans in control. Redub automates repetitive production work while clearly flagging the lines that still need creative judgment.

What is next for Redub

Next, we want to add automatic dialogue separation when a clean music and effects track is unavailable, richer scene context for translation, improved lip synchronization, collaborative review tools, and support for more languages and voice providers.

Our long-term vision is a complete localization studio where creators can turn one episode into many high-quality, verifiable language versions from a single workflow.