Dub Pilot

Inspiration Cinema is supposed to be visual storytelling, but watching a foreign film usually turns into reading a transcript at the bottom of the screen. You spend ninety minutes staring at subtitles instead of watching the actors' faces.

The alternative has always been worse. Traditional dubbing is expensive, takes months, and often strips out the original background audio. Even modern automated dubs usually sound robotic and fall out of sync with the actor's lips.

We wanted to fix this split-attention problem. With multimodal models and Alexa+, we asked a simple question: what if you could just talk to your movie, switch languages on the fly without pausing, and understand foreign idioms without reaching for your phone? That idea became DubPilot.

What it does DubPilot is a multimodal viewing copilot for streaming video. Instead of clicking through buried audio menus, you talk to an Alexa+ agent or use on-screen controls:

Instant language switching: Say "Alexa, switch to Hindi" or tap a chip. The spoken dialogue changes immediately while the video keeps playing at the exact millisecond. Acoustic stem separation: Dialogue runs on its own audio bus. When you change languages, the original background score and sound effects stay intact, ducking naturally under the spoken lines. Sonic etiquette: If the movie is playing dialogue, Alexa will not talk over the actors. It logs its reasoning silently in the chat feed. Alexa only speaks out loud when you pause the video. Cultural idiom explanation: When a culture-specific joke or phrase plays, you can ask Alexa what it means. The copilot uses Claude 3.5 Sonnet on AWS Bedrock to explain the scene context in plain words. Adaptive subtitles: Viewers can resize captions, toggle bilingual text, or adjust contrast with conversational voice commands. How we built it We built the core web application using Next.js 14 (App Router) and TypeScript.

The media engine uses HTML5 video combined with the Web Audio API. We separated the audio into two parallel pipelines: a localized dialogue stem and a background music and effects (M&E) track. This lets us switch spoken languages instantly and apply sidechain ducking without video re-buffering.

For the agent runtime, we built Next.js API routes that connect to the Amazon Bedrock Converse API using Claude 3.5 Sonnet. We wrote custom JSON function-calling schemas for playback control, language switching, subtitle styling, and scene inquiries.

For the localization pipeline, we processed clips through Faster-Whisper for timestamped diarization, aligned target translations to match original syllable counts, synthesized neural audio, and mastered the stems using FFmpeg with EBU R128 loudness standards.

The interface includes a Cinema Darkroom mode for playback and an architectural Monograph showcase highlighting our design tokens and typography.

Challenges we ran into Bedrock streaming tool call fragmentation: When streaming tool invocations through the Bedrock ConverseStream API, parameter payloads arrived as small character deltas that sometimes split multi-byte UTF-8 characters across chunk boundaries. Parsing incoming JSON early broke the stream, so we had to build an accumulator buffer that verifies schemas only on contentBlockStop. Voice assistant audio collision: Early prototypes felt clumsy because Alexa would answer out loud while characters were still speaking. We had to write a playback coordinator that checks audio bus velocity before synthesizing speech. Cross-stem drift in Web Audio: Keeping separate dialogue and ambient audio stems locked to the video playhead during rapid seeks and language toggles caused subtle phase drift. We resolved this by tying gain nodes to a single hardware clock and synchronizing currentTime offsets on every playback state change. Roundtrip voice latency: A standard cloud roundtrip (speech recognition to Bedrock to tool execution) takes over a second, which feels sluggish when watching a video. We added client-side speculative routing for common playback commands, changing the player state immediately while Bedrock runs background verification. Accomplishments that we're proud of Achieving truly gapless track switching. The video never flashes black, pauses, or skips a beat when switching from English to Hindi or Spanish. Sub-120ms tool execution times through Amazon Bedrock. Building an audio etiquette system that respects the movie and never speaks over dialogue. Creating a fully working multimodal simulation with voice, touch, live telemetry, and zero placeholder assets. What we learned Voice control for video requires restraint. Knowing when the assistant should stay quiet is just as critical as what it says. Structured tool calling with AWS Bedrock is fast and dependable, but client-side applications need resilient fallback handling when operating in live streaming contexts. Syllable matching matters far more than literal translation. If the translated line does not match the actor's breathing and phrase length, the illusion falls apart. What's next for Dub Pilot Packaging DubPilot as a native skill for Fire TV and Alexa-enabled smart displays. Building an on-demand cloud processing pipeline on AWS Lambda and ECS so users can paste any video link and get multitrack localized stems within minutes. Adding voice timbre matching so the localized dub preserves the original actor's vocal texture and emotional intensity. Expanding our bilingual subtitle mode into a dedicated interactive tool for language learners. 8:52 PM

Built With

• agentic-ai
• ai-agents
• alexa
• alexa-plus
• amazon-bedrock
• amazon-web-services
• aws-sdk
• claude-3-5-sonnet
• css3
• ebu-r128
• faster-whisper
• ffmpeg
• html5-video
• javascript
• lucide-react
• multimodal-ai
• nextjs
• node.js
• react
• speech-to-text
• text-to-speech
• typescript
• web-audio-api
• web-speech-api