SlateParallel

Four agents walk into the edit bay. A global release walks out dubbed, subtitled, reframed.

Comment

Inspiration

Post-production localization for trailers is normally a slow, sequential grind: transcribe, translate, dub, reframe for each platform, then run compliance checks, one step waiting on the last. I wanted to see how much of that could be automated and, more importantly, parallelized treating a trailer's localization like a fan-out/fan-in multi-agent problem instead of a linear checklist a human producer walks through by hand.

What it does

SlateParallel takes a title, an optional master script, and a source video, then fans out to four specialized worker agents running concurrently:

  • Script & Subtitle Supervisor: transcribes dialogue from the video's actual audio via Gemini's multimodal understanding (or uses a supplied script), broken into per-line timestamped segments with detected speaker gender, not a single flat block of text.
  • Sound Stage & Dubbing Lead : synthesizes a dubbed voice track via Google Cloud Text-to-Speech, placing each line at its own timestamp and matching a male/female voice to the detected speaker.
  • Smart Reframing Director: generates crop/reframe pipelines for vertical (TikTok) and square (Instagram) delivery.
  • Global Standards & Compliance Guardian : queries the Parallel Task API for live, open-web research on regional rating boards (MPAA/BBFC, CERO/Eirin, ICAA, CNC, FSK) and censorship guidelines for each target market.

A fan-in synthesis step (Vertex AI Gemini) compiles all four outputs into a single Master Release Package QC report. A separate final-assembly stage uses Google Cloud Transcoder to actually render the localized video, cropped per platform, subtitles hard-burned in, and the dub audio mixed in so it replaces only the original dialogue , the original music and ambience keep playing underneath, silenced only during the exact windows where dialogue occurs.

How I built it

FastAPI backend on Cloud Run, React/Vite frontend on Vercel. The four workers run via asyncio.gather for genuine concurrency, not just async-flavored sequential calls. Google Cloud Storage stages every intermediate artifact (uploads, dub audio, subtitles, ducked audio, final renders) so a stateless, horizontally-scaled Cloud Run service can hand work between requests without depending on local disk. Firestore persists render-job status for the same reason, polling from a different instance than the one doing the work still needs to find the job.

Challenges I ran into

  • Cloud Run is stateless and ephemeral: The first version wrote worker outputs to local disk; a later request could land on a different instance and find nothing there. Fixed by staging everything through GCS and Firestore instead of assuming one process's memory/disk persists.

  • Transcoder API has real, undocumented-until-you-hit-them constraints: It rejects AV1 input entirely, can't mux a standalone text/caption stream into a plain mp4 (subtitles have to be hard-burned into pixels instead),and critically for the dialogue-removal feature indexes video and audio as one shared track space per input rather than separately, which cost us a debugging session before AudioMapping.gain_db told us it actually could mix multiple audio sources at different volumes at all.

  • Gemini's JSON output had no fixed schema by default. : Asking for "a structured JSON report" got a different field set almost every call (overall_status vs overall_qc_status, summary vs qc_summary...), unusable for a UI to bind to reliably. Fixed by constraining output with Vertex AI's response_schema which, undocumented until you hit it, requires uppercase type names ("OBJECT", "STRING") rather than JSON-Schema's lowercase; the first attempt failed with a live KeyError: 'object'.

  • Gemini's Files API can't reference a gs:// object directly : That's a Vertex-AI-SDK-only capability; the plain API-key client needs the bytes downloaded and uploaded through files.upload instead.

  • Subtitle/dub timing wasn't actually synced to the video at all: Originally, an early placeholder just showed the whole translated block for a fixed few seconds. Getting genuinely per-line timestamped subtitles and correctly time-placed dub audio (instead of one fused clip starting at 0s) was the single biggest quality improvement in the whole project.

Accomplishments that I am proud of

  • Real per-line timestamp sync end to end, Gemini-detected (or script-specified) dialogue timing flows through translation, subtitle cues, and dub-audio placement consistently, verified against actual rendered output, not just inspected in a JSON response.
  • Discovering and using Transcoder's native multi-source audio mixing to silence only the original dialogue while leaving music/ambience playing underneath, confirmed with real volume measurements on a rendered file (full volume outside dialogue windows, true digital silence inside them).
  • Gender-matched dubbed voices, detected automatically from the source video's own audio rather than defaulting to one fixed voice for every line regardless of who's speaking.
  • A render pipeline that survives Cloud Run's stateless, multi-instance reality instead of quietly breaking the moment a request lands on a different instance than the one that started the work.
  • I didn't just trust that things worked, nearly every fix in this project was verified against live infrastructure (real Transcoder jobs, real Gemini calls, real GCS uploads), which caught several bugs that would have looked fine in code review alone.
  • A real concurrency benchmark, not just a claim; I measure actual fan-out wall-clock time against a genuinely equivalent sequential estimate for the same four workers and inputs, so "parallel, not sequential" is backed by a number on screen, not an assumption, and I made sure that number was honest.

What I learned

Engineering this pipeline reinforced that runtime execution regularly disproves local assumptions. Submitting live jobs revealed that Google Cloud Transcoder cannot mux text tracks into plain MP4s, demanding an upstream architectural shift to FFmpeg caption burn-ins. Downstream encoding broke on unexpected AV1 streams, proving that container extensions are never a substitute for strict codec validation. When third-party extraction succeeded locally but was blocked from Cloud Run datacenters, scoping out the feature proved far more sustainable than deploying brittle, high-risk authentication hacks. Finally, serverless deployments completely broke local-disk assumptions: multi-instance scaling meant subsequent requests landed on separate instances that lacked previous memory or local outputs, forcing an immediate transition to centralized state management and asset staging via Google Cloud Storage and Firestore.

What's next for SlateParallel

  • True ML-based vocal isolation, so background music can keep playing underneath dialogue too, instead of being silenced for the same window as the voice it can't yet be separated from.
  • A more resilient path to source video beyond direct file URLs/uploads, revisiting YouTube ingestion via authenticated cookie-based access, or supporting more source platforms generally.
  • Per-character voice consistency across an entire video, not just per-line gender matching, the same speaker should keep the same voice throughout, not just "a male voice" or "a female voice" each time.
  • Finer-grained render progress instead of just PENDING/SUBMITTED/SUCCEEDED/FAILED, since Transcoder itself exposes no percentage-complete signal today.
  • CI/CD for the backend to match what the frontend already gets for free, Vercel auto-deploys on push; Cloud Run currently still needs a manual gcloud run deploy after each change.

Built With

Share this project:

Updates

Submission history