LOCALIZE

Inspiration

Most automated dubbing tools treat localization like an assembly line: audio gets transcribed to text, translated by a generic LLM, pushed through text-to-speech, and slapped back onto video. If a translated phrase takes two seconds longer to speak than the original actor's mouth movement, or if an idiom translates literally into nonsense, the pipeline still outputs the file without checking the result.

Human dubbing studios do not work this way. Directors, script translators, voice actors, sound engineers, and QA reviewers work together. When a translated sentence runs too long, the team rewrites the line or adjusts speech cadence to fit the dialogue window before publishing. We wanted to build an autonomous crew that behaves like a real post-production room, complete with automated quality control and targeted repair loops.

What it does

LOCALIZE takes an input video (MP4 or MKV) and a target language or cultural tone prompt, then turns it into a synchronized, localized dub.

Instead of running a one-way script, it coordinates seven specialized agents:

1. Director Agent: Manages state, coordinates handoffs between agents, and handles targeted retries when defects are caught.
2. Story Analyst: Maps out characters, extracts scene boundaries, tags emotional tone, and spots cultural idioms.
3. Localization Director: Translates dialogue while preserving character personality and jokes, writing down the translation rationale for every line.
4. Voice Director: Matches distinct character profiles to neural TTS voices and generates clean speech stems.
5. Sync Engineer: Aligns audio timing to the original dialogue windows using dynamic FFmpeg atempo speed adjustments and millisecond offset shifts.
6. Subtitle Director: Generates timestamped SRT and WebVTT subtitles that match the newly synchronized audio.
7. QA Continuity Agent: Audits the resulting cut for timing overflows, subtitle drift, and clipping. If it finds an issue, it sends instructions back to the specific upstream agent to fix that segment without restarting the entire run.

Users can watch the pipeline execute in real time on a dual-monitor studio interface, compare before and after cuts, and inspect agent telemetry.

How we built it

• Multi-Agent Reasoning: Built using Google Gemini structured JSON outputs. Structured schema enforcement ensures reliable data contracts between the Story Analyst, Localization Director, Sync Engineer, and QA Agent.
• Audio and Video Engine: Python, FFmpeg, and FFprobe handle stream demuxing, precise millisecond audio slicing, atempo filter chaining, and vocal track replacement.
• Backend Service: FastAPI with async background job execution, SSE (Server-Sent Events) for live pipeline progress, and Pytest suites for unit and integration testing.
• Frontend Studio: Next.js (App Router), TypeScript, and Tailwind CSS. The interface includes an agent sequence track, an interactive timeline player, a before/after split viewer, and a live QA defect log.
• Observability: Grafana MCP endpoints and telemetry trackers capture agent latencies, retries, and quality scores across each run.

Challenges we ran into

• Syllabic Expansion and Timing Windows: Some languages naturally use 20% to 35% more syllables to express the same thought. Making the audio fit required a combination of upstream script rewriting and downstream atempo time-stretching, without making character voices sound artificially rushed.
• Targeted Self-Repair: Building a feedback loop that only reruns the affected dialogue segment instead of recomputing the entire video pipeline required isolating intermediate agent artifacts (scripts, voice stems, sync maps) by dialogue ID.
• Real-Time Studio Telemetry: Streaming intermediate agent thoughts, audio generation states, and video render events to the browser without UI lag meant fine-tuning our SSE event dispatchers and state reducers.

Accomplishments that we're proud of

• Autonomous QA Feedback Loop: When the QA agent flags a timing overflow defect, it automatically loops back to the Localization Director or Sync Engineer to re-adjust and re-render only the offending line.
• Character Voice Consistency: Distinct speakers in a scene receive consistent neural voice assignments across every cut.
• Sub-Millisecond Sync Precision: Clean alignment between vocal delivery, background audio mixing, and subtitle generation.
• Complete Test Coverage: Built with test-driven workflows across both backend engines and frontend components.

What we learned

• Structured JSON constraints on LLMs are essential when chaining multiple reasoning steps. Loose text prompts break pipelines; strict schemas keep agents dependable.
• High-quality dubbing is as much an audio engineering challenge as a translation challenge. Timing, pauses, and background sound preservation matter just as much as word choice.
• Real-time observability makes debugging agent decisions significantly faster during active development.

What's next for LOCALIZE

• Lip-Sync Motion Synthesis: Integrating neural video face-warping models to align visual mouth movements with adjusted speech tracks.
• Multi-Track Background Isolation: Direct integration of Demucs separation to preserve complex background ambient audio and musical scores automatically.
• Team Collaboration Mode: Letting human dubbing directors approve, override, or tweak individual agent suggestions before the final master render.

Built With

• css
• fastapi
• ffmpeg
• gemini
• google
• grafana
• next.js
• pytest
• python
• tailwind
• typescript