ai-video-translation

AI-powered video localization that transforms Chinese videos into natural English versions with synchronized voiceovers, subtitles, and platform-ready exports.

Comment

About the Project

Inspiration

Many great Chinese story videos never reach international audiences because localization involves much more than literal translation. The narration must sound natural, fit the original timing, preserve background audio, and remain synchronized with subtitles and visuals.

This inspired me to build an AI-assisted video localization workflow that turns Chinese videos into platform-ready English content while keeping the original picture and atmosphere intact.

What It Does

The project processes a Chinese video through a structured workflow:

  1. Upload and preview the source video.
  2. Separate the Chinese voice from music and environmental sounds.
  3. Transcribe the narration with ASR.
  4. Reconstruct fragmented ASR lines into complete Chinese sentences.
  5. Rewrite each sentence as natural, conversational English.
  6. Generate English speech and validate its duration against the available time slot.
  7. Create synchronized subtitles, mix the new narration with the original background audio, and export finished videos.

The final package can include a standard YouTube video, a TikTok version, and vertically formatted YouTube Shorts.

How I Built It

The backend is built with Python and FastAPI. FFmpeg handles audio extraction, mixing, subtitle rendering, video composition, and platform-specific exports.

The AI workflow uses:

  • Volcengine AI MediaKit for audio processing and speech recognition
  • Doubao Ark models for semantic sentence reconstruction and natural English rewriting
  • Kokoro TTS for local English voice generation
  • A checkpoint system that preserves completed work and resumes failed tasks

The frontend provides a step-by-step review workflow. Users can preview each result, inspect sentence timing, retry individual lines, and approve the final output.

For every sentence, the system compares the generated speech duration with its allowed interval:

[ d_{\min} \leq d_{\text{speech}} \leq d_{\max} ]

When multiple versions exist, it selects the valid candidate closest to the target duration. If none fit perfectly, it prefers an under-length version over one that overlaps the next sentence.

Challenges

The hardest problem was synchronization. Chinese and English express the same idea with different rhythms, so direct translation often creates English narration that is too long, too short, or unnatural.

Other major challenges included:

  • Rebuilding complete sentences from fragmented ASR subtitles
  • Avoiding literal, unnatural Chinese-to-English phrasing
  • Preserving pauses without creating overlapping speech
  • Preventing repeated TTS requests from increasing cost
  • Recovering from network and API failures without restarting the workflow
  • Keeping audio, subtitles, diagnostics, and previews based on the same final candidate
  • Exporting one source video into different platform formats

What I Learned

I learned that high-quality localization is not a single AI request. It is a coordinated media pipeline involving transcription, semantic reconstruction, adaptation, speech synthesis, timing analysis, and video engineering.

I also learned that the best results come from combining AI with deterministic code: AI handles language and meaning, while Python and FFmpeg enforce timing, caching, file management, and reproducible exports.

The result is a practical production tool that makes multilingual video localization faster, more transparent, and easier to control.

Built With

Share this project:

Updates