Agentic AI Dubbing System

Self-healing multi-agent AI dubs Indian cinema with perfect lip-sync & emotion — where scripts fail silently, our agents catch errors, learn, and fix themselves. Next: seamless visual lip-sync.

Comment

Inspiration

Regional Indian cinema is full of great performances lost in translation — literally. Dubbed versions suffer from robotic voices, mismatched lip-sync, and a total loss of the original actor's emotional delivery. We noticed that every existing automated dubbing script shares the same silent flaw: it blindly passes errors down the line. A noisy Kannada line gets mistranscribed, confidently mistranslated, and fluently spoken as complete nonsense — and the pipeline never knows it failed. We wanted to build something that actually catches its own mistakes.

What it does

IndicDub is a multi-agent dubbing pipeline coordinated by a central FastOrchestrator. It takes an audio input, transcribes it, translates it, and generates natural-sounding dubbed speech — but unlike a linear script, every stage grades its own output:

  • An ASR Agent (OpenAI Whisper) transcribes speech and checks its own confidence score, automatically escalating to a larger model if it's unsure.
  • A Translation Agent (Groq LLaMA-3.3-70B) translates using one of four strategies (natural, literal, formal, simplified) and flags translations that are too short, too long, or contain leftover source-language text.
  • A Voice Agent (Microsoft edge-tts) synthesizes speech across 14 language voices and checks for silence or corrupted audio before accepting the output.

A persistent memory store tracks which translation strategy scores best for each language pair, so the system gets smarter — and faster — with every job it completes.

How we built it

We built the system around a FastOrchestrator that coordinates three specialist agents, each with its own self-evaluation logic and scoring rubric (ASR 30%, Translation 35%, Voice 35% of the final score).

  • Audio is normalized to 16kHz mono WAV using pydub.
  • The ASR Agent runs OpenAI Whisper and evaluates its average log-probability, language match, and words-per-second ratio to detect hallucination or low-confidence transcription.
  • The Translation Agent calls Groq's LLaMA-3.3-70B API, checks word-count ratio and script/language leakage, and caches repeated phrases to save API calls.
  • The Voice Agent uses Microsoft's edge-tts for free, high-quality neural speech synthesis, checking output duration and RMS silence before accepting a result.
  • All of this is wrapped in a persistent memory store (agent_memory.json) that logs every job and learns the best-performing translation strategy per language pair over time.

The whole pipeline runs in Google Colab for the hackathon prototype.

Challenges we ran into

Getting agents to fail gracefully was harder than getting them to succeed. Defining the right thresholds — how low is "too low confidence" for Whisper, how short is "too short" for a translation — took a lot of tuning against real regional-language audio. Balancing self-eval strictness against pipeline speed was a constant trade-off: retrying too aggressively slows everything down, but retrying too little lets bad output slip through.

Accomplishments that we're proud of

We built a system that catches its own failures instead of silently propagating them — the exact problem that makes most automated dubbing tools unreliable for real cinematic content. The memory-guided strategy selection means the system doesn't repeat the same mistakes across jobs; it actually learns which approach works best for each language pair.

What we learned

We learned that "agentic" isn't just about chaining API calls — it's about giving each step the ability to judge its own output and make a decision. We also learned a lot about the specific translation challenges of Indian regional languages: syllable density, idiom preservation, and code-switching all break naive translation approaches in ways a single confidence check can't catch.

What's next for Agentic AI Dubbing System

Next, we're integrating visual AI — automated face tracking and lip-syncing (using LivePortrait or VideoReTalking) to manipulate video frames directly, so the mouth movements match the new dubbed audio. This closes the loop from audio-only dubbing to a fully seamless visual dubbing experience.

Built With

Share this project:

Updates

Submission history