Dhvani - "Watch global. Listen local".

YouTube hosts 14B+ videos with 500 hours uploaded every minute and 60% is non English. Dhvani lets you experience any content in your language in a real time natural native voice.

Comment

Inspiration

YouTube has 14 billion videos. 500 hours of new content uploaded every minute. Yet 60% of it is locked behind a language barrier. Subtitles exist, but reading while watching kills the experience. Existing dubbing tools are slow, robotic, and built for studios, not for the person who just wants to watch a video right now. We asked: what if you could press play on any YouTube video and hear it in your language, in the speaker's own voice, instantly? That question became Dhvani.

What it does

Dhvani dubs any YouTube video in real-time. Paste a URL, pick a language, Spanish, Japanese, or Chinese, and Dhvani streams back the audio dubbed in the original speaker's cloned voice, synchronized to the video as it plays. It preserves the speaker's emotion and tone, not just their words.

How we built it

  1. Paste any YouTube URL, Dhvani downloads the audio, splits it into 3-second chunks and opens a WebSocket connection
  2. Each chunk flows through a 3-stage concurrent pipeline: Higgs ASR 3 transcribes the audio, GPT-OSS translates the transcript, and Higgs TTS 2.5 generates dubbed audio with voice cloning
  3. All three stages run in parallel so there is no waiting. While TTS speaks chunk N, GPT-OSS translates N+1, Higgs ASR 3 transcribes N+2
  4. Voice reference is picked from the loudest of the first 5 chunks and cloned across all dubbed audio using Higgs TTS 2.5
  5. YouTube captions are fetched at load time and used to validate transcriptions, catching hallucinations before they reach TTS
  6. Frontend uses YouTube IFrame API with timestamp-aware playback. Each chunk knows its exact video position, late chunks are discarded, early chunks wait
  7. Built on FastAPI with WebSockets, with a disk cache so repeated language requests play back instantly

Challenges we ran into

  1. Real-time sync: processing a 3-second chunk takes longer than 3 seconds, so dubbed audio is always chasing the video. Solved with per-chunk timestamp sync anchored to the video clock
  2. Higgs ASR 3 latency was bimodal, fast calls at around 1 second and slow calls hitting 5 to 8 seconds during cold starts. Solved by running ASR once per chunk and sharing the transcript across all language pipelines
  3. TTS hallucinations: Higgs TTS 2.5 with voice cloning occasionally generated 57 seconds of English audio for a 3-second chunk. Fixed with YouTube caption validation as a ground truth filter
  4. Queue management: without bounded async queues, the fast ASR and translation stages piled up work faster than TTS could consume it, causing growing delays across the pipeline

Accomplishments that we're proud of

• Built a fully working real-time dubbing pipeline from scratch in under 48 hours • Voice cloning that sounds like the original speaker, preserving emotion and tone across languages • True real-time sync where dubbed audio plays exactly at the right moment in the video, not just sequentially • Transcription shared across all language pipelines so the same audio chunk is never processed twice • A system that goes from any YouTube URL to dubbed audio in under 10 seconds, with no GPU required

What we learned

• Real-time audio sync has to be anchored to the video clock, not the processing clock. The moment you decouple those two, everything drifts • Concurrent pipelines only work if every stage is bounded. Unbounded queues let fast stages silently bury slow ones • Voice cloning quality depends on reference selection. The loudest chunk, not the first one, produces the cleanest clone • TTS hallucinations are real and unpredictable. Ground truth validation from an external source is the only reliable fix • Latency spikes in cloud APIs are more damaging than higher average latency. Consistency matters more than speed for real-time systems

What's next for Dhvani - "Watch global. Listen local".

• Support for 5+ languages including Hindi, Korean, French • Browser extension so Dhvani works on any YouTube video without copy-pasting URLs • Mobile app for on-the-go dubbing of YouTube Shorts and long-form content • Creator tools so YouTubers can publish pre-dubbed versions of their videos in multiple languages with one click

Built With

Share this project:

Updates

Submission history