Inspiration
Most dubbing tools translate words. They do not translate why a scene works.
A joke can become confusing, an idiom can sound unnatural, a celebrity reference can lose its meaning, and a perfectly translated line can still fail because it no longer fits the actor’s speaking time. This is especially visible when adapting English video for Indian audiences, where language, region, rhythm, and cultural context all matter.
We built LocalCut as an AI cultural adaptation director, not merely a translator. It helps creators turn an English scene into a culturally fluent Hindi or Kannada version while preserving its tone, timing, intent, and energy.
What it does
A creator uploads a short English video, and LocalCut runs an end-to-end adaptation workflow:
- Transcribes the dialogue with speaker labels and timestamps.
- Analyses emotion, delivery, pacing, and tone directly from the audio.
- Detects jokes, idioms, celebrities, locations, products, and cultural references that may not transfer naturally.
- Uses Parallel Search to research relevant Indian, Hindi, Kannada, and Karnataka-specific context.
- Produces a citation-backed adaptation brief explaining its choices.
- Rewrites each line to preserve meaning while fitting approximately the original speaking duration.
- Generates a timed, multi-speaker Hindi or Kannada voice preview.
- Identifies visible text, signage, products, landmarks, or shots that may also require localization.
- Lets the creator review and edit every line before approving the dub.
- Renders a downloadable MP4 with mixed audio and translated subtitles.
The result is not just translated dialogue. It is an editable creative direction package backed by audio evidence, cultural research, timing measurements, and human approval.
How we built it
The frontend is a responsive adaptation studio built with Next.js, React, TypeScript, and Tailwind CSS. It begins with a distraction-free upload experience and reveals the complete review dashboard only after the pipeline finishes.
The backend runs as a FastAPI service on Google Cloud Run. Videos are uploaded directly from the browser to Google Cloud Storage using short-lived signed URLs. This avoids transferring large media files through Vercel or ordinary API request bodies.
Google Agent Development Kit coordinates specialized analysis and adaptation stages. Gemini 2.5 Flash handles multimodal understanding, including dialogue, speakers, acoustic tone, and visible elements. Gemini 2.5 Pro performs the deeper cultural rewrite and timing-aware adaptation.
Parallel Search supplies current cultural research for references that require localization. LocalCut retains the relevant sources and presents them alongside its recommendations so creators can inspect why a change was made.
Google Cloud Text-to-Speech generates Hindi and Kannada dialogue using Chirp 3 HD voices. LocalCut measures the generated speech, adjusts timing, assembles multiple speakers, mixes the dub with the original ambience, and uses FFmpeg with libass to render the final video and subtitles.
Firestore stores project state, Cloud Storage holds temporary media, Secret Manager protects credentials, and lifecycle rules remove uploaded media after 24 hours.
Challenges we faced
The hardest challenge was keeping the adaptation grounded in the original performance. Analysing text alone misses sarcasm, hesitation, excitement, and comic timing. We therefore incorporated acoustic evidence from the source audio into each adaptation decision.
Speaking duration was another challenge. A natural translation is often considerably longer or shorter than the English line. We created a measured preview loop that synthesizes each line, checks its actual duration, and adjusts its delivery to fit the original dialogue window.
We also had to handle two very different writing systems and cultural contexts. Hindi adaptation cannot simply be reused for Kannada. The Kannada workflow needs Karnataka-specific research, language patterns, references, and voices of its own.
Finally, video processing required a different architecture from a normal web application. Direct Cloud Storage uploads, signed URLs, bounded media validation, Cloud Run processing, FFmpeg rendering, subtitle support, and browser CORS all had to work together reliably.
Accomplishments that we are proud of
We built a complete path from an owned English video to an editable, citation-backed Hindi or Kannada adaptation and a rendered multi-speaker MP4.
We are especially proud that LocalCut exposes its reasoning instead of hiding it. Creators can compare the original line, literal translation, localized recommendation, acoustic evidence, cultural flags, research sources, measured timing, voice choice, and visual replacement guidance in one workspace.
The human approval checkpoint also matters to us. LocalCut provides direction and production assistance, while the creator retains control over the final script and generated voices.
What we learned
We learned that high-quality localization is a multimodal problem. Language models, audio understanding, live research, speech synthesis, and deterministic media tooling each solve a different part of it.
We also learned that cultural adaptation works best when AI recommendations are inspectable. Citations, timing reports, and editable decisions make the system more useful and trustworthy than an opaque translate-and-generate button.
What’s next
Next, we want to add more Indian languages, stronger regional voice selection, automatic speaker-to-voice matching, improved lip and pause alignment, collaborative review, and frame-level replacement previews for localized signs and products.
Our long-term goal is to make culturally fluent video adaptation accessible to every creator, not only studios with large localization teams.
Log in or sign up for Devpost to join the conversation.