Ajilamu: Agentic Precision Dubbing & Ledger

Agentic video dubbing meets ClickHouse. AI agents repair speech timing while an immutable columnar ledger tracks every edit and API fee.

Comment

Inspiration

Six centuries ago in western Arunachal Pradesh, the Monpa people created Aji Lhamu. It is a masked dance-drama carrying the Tibetan Ramayana across languages and borders.

The tradition stems from Thangtong Gyalpo, the fifteenth-century engineer, poet, and dramatist. He staged performances to finance 108 iron suspension bridges across treacherous Himalayan gorges.

Two lessons from that history anchor this project.

First, a masked player cannot rely on facial expression. The mask fixes the face, so the spoken line must land on physical movement exactly. That is the video dubbing problem, six centuries early.

Second, theatre paid for bridges. Art carrying stories across languages funded crossings that carried people. Software translating film between languages sits in that exact line of work.

Modern AI dubbing tools routinely break this contract. They generate speech, stretch audio clumsily, or let voices talk over closed mouths. We built Ajilamu to treat dialogue duration as an unyielding physical constraint.

What It Does

Ajilamu takes creator video and a target language. It returns a studio-grade dubbed video and an itemized audit ledger.

  1. Watch: Gemini 3.8 Flash inspects the video. It extracts timestamped dialogue boundaries, speaker names, and emotional register.
  2. Translate under a budget: Gemini translates each line with slot duration as a strict mathematical boundary, not a suggestion.
  3. Render: Google Cloud Chirp 3 HD synthesizes the line using voice casting mapped to the documented gender of the speaker.
  4. Measure: ffprobe reads the resulting WAV file. It computes the signed duration delta against the physical slot.
  5. Repair: A deterministic two-sided fit loop corrects discrepancies.
  6. Assemble: ffmpeg overlays discrete takes onto the original audio bed, ducks background music, and muxes the master video.

The non-dialogue audio bed survives everywhere outside speech slots. Music, room tone, and Foley sound remain intact. The export retains the original sample rate and channel layout without downsampling.

Every take remains a discrete WAV file until final export. Creators can solo lines, preview historical takes, and edit non-destructively.

How We Built It

Ajilamu combines a Go server, a Svelte 5 frontend, and an append-only ClickHouse ledger:

  • Go 1.27.1 Backend: Standard library first. The Go server coordinates pipeline execution, media transformations, and durable database writes.
  • ClickHouse Cloud Ledger: ClickHouse stores all project state across five raw append-only tables (takes_raw, commits_raw, timeline_state_raw, actions_raw, and charges_raw). The system forbids in-place updates. Views reconstruct timeline state at any historical commit, enabling instant branching and rollback.
  • Waveform Precomputation: Peak arrays live in ClickHouse as Array(UInt8). The frontend visualizes waveforms immediately upon page load without downloading multi-megabyte audio files.
  • Agentic Editing via ADK & MCP: An AI editor agent built with Google ADK (adk-go) interprets natural language commands. It queries the ledger over a self-hosted mcp-clickhouse Docker container. The Go backend acts as the sole database writer, while the agent reads through read-only MCP.
  • Media Engine: Native ffmpeg n9.0.1 filters execute audio processing: atempo for speed adjustments, sidechaincompress for dynamic music ducking, and amix for stem balance.
  • Svelte 5 Workspace: Built with modern Svelte 5 runes ($state, $derived, $props). The interface features length bars that color-code timing precision and stream real-time progress via Server-Sent Events.
  • Production Infrastructure: Rebuildable single-host deployment on Google Cloud. Caddy terminates TLS, while internal services bind loopback. Google Cloud Secret Manager injects secrets into tmpfs at /run/ajilamu/env.

Challenges We Faced

1. The Two-Sided Fit Trap

Traditional dubbing software only flags overrun errors. On 2026-09-07, our baseline script evaluated a line that finished 40.9 percent short of its slot. The script printed "perfect fit" and zero defects. In playback, 2.9 seconds of dead air played over moving lips.

We solved this by formalizing fit as a signed delta:

$$ \Delta t = t_{\text{measured}} - t_{\text{slot}} $$

The signed delta dictates the repair strategy:

  • Within margin ($|\Delta t| \le 0.08 \cdot t_{\text{slot}}$): ffmpeg atempo stretches or compresses audio without altering pitch. This costs zero API tokens.
  • Overrun ($\Delta t > +0.08 \cdot t_{\text{slot}}$): Gemini rewrites the line with concise vocabulary.
  • Underrun ($\Delta t < -0.08 \cdot t_{\text{slot}}$): Gemini expands the line with richer phrasing.
  • Cap: After 3 attempts, the line flags for creator review in the workspace UI.

2. Nanodollar Cost Accounting

AI agents can trigger runaway API costs through silent retries. We priced every operation in nanodollars in internal/cost/cost.go:

  • Chirp 3 HD synthesis: \$30.00 per million characters
  • Gemini prompt tokens: \$0.15 per million
  • Gemini completion tokens: \$0.60 per million

Every rejected repair attempt logs its own row in charges_raw. A 75-second NASA sample dubbed into Malayalam measured exactly \$0.023414. Nothing hides inside the winning take.

3. Audio Bed Preservation

Demuxing mixed video often degrades sound quality. We isolate the 16 kHz mono demux exclusively to Gemini analysis. The final assembly mixes freshly rendered dialog takes directly over the untouched original soundtrack.

What We Learned

  1. A pin is a measurement, never a log line: Software declaring its own success is the object under review, not evidence for it. True validation requires external instruments (ffprobe, database assertions, DOM inspection).
  2. Immutability simplifies agentic collaboration: When agents and humans edit the same timeline, an append-only ClickHouse DAG eliminates merge conflicts.
  3. Runes make reactive media interfaces clean: Svelte 5 runes kept playback scrubbers, waveform caches, and live WebSocket status synchronised without store spaghetti.

What's Next for Ajilamu

  • Visual Speaker Diarization: Aligning speaker identity with face recognition when multiple people speak in one scene.
  • On-Screen Text Translation: In-painting translated signage and captions directly onto video frames.
  • Community Dialect Voice Cloning: Expanding voice casting to endangered Himalayan and regional languages.

Built With

Share this project:

Updates