Ad Localisation for Global Market

OmniSwarm turns one brand video into native-feeling global versions, using AI agents to localize voice, visuals, culture, and compliance for every market.

Comment 1

Inspiration

We kept seeing the same problem: a brand spends a lot on one ad, then waits weeks (and pays again) to adapt it for Japan, Germany, India, and everywhere else. New voiceovers, subtitles, compliance checks, all spread across different tools and vendors. A lot of teams end up shipping the same English creative globally because proper localization feels too slow and too expensive.

Ad Localisation for Global Market (OmniSwarm) came from a simple idea. What if you could upload one master ad, pick your markets, and get localized versions in minutes, on a stack you could actually ship with, not just demo for a weekend?

What it does

OmniSwarm is a full-stack ad localization platform. You upload a master video and voiceover, or use our built-in sample creative. Pick your target markets and launch a campaign.

The dashboard shows everything as it runs: security checks, isolated working environments per market, video and audio agents doing their work, and final assembly. When it's done, you can play the localized ads right in the browser using secure S3 presigned URLs.

Each market gets its own language, voice, and regional styling. The important part: these are real translated voiceovers (Japanese, German, Hindi, English). We are not copying the same English audio four times and calling it localized.

For the hackathon demo, judges can hit Run Instant Demo on the live app, watch all markets finish in seconds, and compare the finished videos side by side.

Live app: https://omniswarm-h0.vercel.app

How I built it

This project uses the H0 hackathon stack end to end.

The frontend and API are Next.js 14 (App Router, TypeScript, Tailwind) deployed on Vercel. Job state lives in Amazon Aurora DSQL in a localization_jobs table. Markets, forks, agents, results, and logs are stored as JSON so the Node app and Python worker stay in sync. Media files go in Amazon S3 (master bucket + output bucket), with presigned URLs so the browser uploads and streams directly without heavy files passing through Vercel.

The Vercel app creates jobs, polls status, and runs a serverless localization pipeline. The heavier work (FFmpeg remuxing, Gemini translation, text-to-speech per market) lives in an optional Python FastAPI worker with FFmpeg, Demucs, and Gemini.

For the live demo, we pre-rendered real translations once into a canonical S3 library at campaigns/demo_master/<market>/. When someone runs the demo, the serverless function just copies the right localized bundle per market. That keeps it fast on Vercel while still showing genuinely translated output.

Both the web app and worker read and write the same DSQL schema and S3 buckets. We use APP_AWS_* env vars instead of AWS_* because Vercel reserves those. DSQL connects with IAM auth from Node (pg + @aws-sdk/dsql-signer) and Python (psycopg + boto3 tokens).

Challenges I ran into

AWS permissions took a while. Early provisioning failed until we attached the right DSQL and S3 policies to the IAM user.

Vercel can't do the heavy media work. FFmpeg, Demucs, and long Gemini calls don't fit in a serverless function. We split it: instant orchestration on Vercel, real transcoding in the Python worker when needed.

Our first pipeline faked translation. The self-contained version copied English audio to every market and only logged "Gemini dub" in the UI. Fixing that meant wiring real translation, and for the public demo, pre-rendering once into S3.

Gemini quotas are real. The free tier TTS limit (10 requests per day per model) blocked us from rendering the full library in one go. We reused earlier successful outputs and used macOS Hindi TTS for India when we hit the limit.

ffmpeg broke on my Mac. Homebrew's default ffmpeg@6 was missing libvpx, so dubbing failed quietly. We added an FFMPEG_BIN override and auto-detection in run.sh.

Videos kept restarting during playback. Polling job details regenerated presigned URLs every couple seconds, which reset the video player. We stopped polling once a job hit completed or failed.

Speed vs. authenticity. Real worker translation takes about 60 to 90 seconds per market, so four markets can take 10+ minutes. The pre-rendered library plus S3 copies let us keep the demo instant without giving up real localized audio.

Accomplishments that I'm proud of

We got a full pipeline running on production-grade AWS and Vercel: Aurora DSQL, S3, and Next.js, with a data model that actually makes sense for job orchestration.

The one-click demo works for judges without any file upload, and the live site returns real per-market voiceovers in about 5 to 10 seconds.

The agent-style orchestration (video and audio parent agents, per-market forks, live logs) feels close to how a real localization studio would parallelize work.

Vercel and the Python worker share job state through Aurora DSQL, so the dashboard always reflects what's actually happening.

The UX feels shippable: dark dashboard, market selection, optional upload, stable video playback, and S3 previews that actually work.

What I learned

Serverless is great for orchestration, not for heavy media. Vercel handles UI, API, and fast state updates. S3 holds the assets. A worker (or a pre-rendered library) handles the CPU-heavy steps.

Aurora DSQL IAM auth needs short-lived tokens on both Node and Python. Plan for token refresh, and pick env var names carefully (APP_AWS_* on Vercel).

Demo architecture matters as much as demo features. Pre-rendering translated assets once and serving them via S3 copies made the hackathon demo both fast and honest. Judges don't wait 10 minutes or hit rate limits mid-demo.

Small UX details add up. Stopping presigned URL churn during playback was a tiny code change but made the videos actually watchable.

What's next for Ad Localisation for Global Market

On-demand real translation for custom uploads, not just the demo library. Trigger the Python worker or a managed GPU service when a user brings their own creative.

More markets and voices, with locale-specific rules (keigo for Japan, GDPR copy checks for EU, and so on).

Better audio mixing with full Demucs stem separation so localized speech sits cleanly over the original music bed.

Batch campaigns to localize whole ad libraries from a CSV or CMS hook.

Quality gates before publish: lip-sync checks, duration validation, and optional human review.

Cost controls so this stays viable at scale: per-job budgets, Gemini quota management, and S3 lifecycle rules.

Built With

Share this project:

Updates