Lenguaraz

Inspiration

Every multi-track conference loses part of its audience in every room: people who don't follow the speaker's language, people who don't hear well, people watching remotely on a bad connection. Human interpreters are wonderful and scarce, and captions on one screen serve one room. We wanted every attendee to open a link on their phone, pick a stage and a language, and read the talk as it happens, in English, Spanish or Portuguese.

The name is Rioplatense Spanish: on the 18th–19th century Río de la Plata frontier, the lenguaraz was the interpreter who stood between peoples who didn't share a language, so everyone in a parley could follow what was said.

What it does

• Listens to the audio every stage already has (SRT from OBS/vMix, RTMP, HLS or a file) and transcribes it live with the Gemini Live API (gemini-3.5-transcribe-live): partial captions while the speaker talks, a glossary of technical terms and proper names as custom_vocabulary.
• Translates each sentence into every language someone is watching with gemini-3.5-flash-lite, glossary-aware and with context.
• Audience page per stage (language picker, font size, high contrast), a transparent OBS overlay, an operator dashboard (state, latency, cost, start/stop) and SRT/VTT/TXT export. The English subtitles of our video were exported from Lenguaraz itself.
• Runs many stages in one process, in Docker, with HTTPS, and a credential-free dry-run mode.

How we built it

Spec-driven development inside the 24-hour window: a written constitution, then spec → plan → tasks → implementation loop for every feature, with an AI engineering crew (Claude Code) and one human owner. Every Gemini API field was verified against the official docs through the Gemini Docs MCP before it was used, and every claim in the README is produced by a script in the repo (make smoke-stt, make simulate, make loadtest, make mvp-check).

Pipeline: ffmpeg → Gemini Live transcription (hybrid VAD, make-before-break session rotation, stall watchdog) → per-language translation workers → WebSocket fan-out → React + Tailwind + shadcn/ui pages. Python 3.12 + FastAPI + asyncio on the backend. CI runs lint, types, tests, gitleaks, Trivy, an SBOM and a license gate, and make fresh-clone-test proves the quickstart works from a clean clone.

Challenges we ran into

1. Translation latency on a congested day. The text model answered in 0.7–2 s for most requests but 11–40 s for some. Three parallel calls came back in 38.3 s, 1.6 s and 1.7 s, so the slowness is per request. We added hedged requests: if a translation hasn't answered after 600 ms, we send an identical copy, and the first answer wins. If a fraction $p$ of requests is slow, the chance that all $k$ copies are slow is

$$P(\text{all } k \text{ slow}) = p^{k}, \qquad p \approx 0.5,\; k = 3 \;\Rightarrow\; 12.5\%$$

and captions are short, so the extra cost is negligible. Together with parallel translation published in order, a head-of-line cap and reuse of the partial translation, the Spanish→English delay went from 5–30 s to a median of 0.6 s, measured live on a Nerdearla talk streamed from OBS.

2. Live transcription behaving differently on demo day. Partials became cumulative, finals arrived only after whole paragraphs, and sessions sometimes went silent. We now close sentences at punctuation (with a 28-word cap), align cumulative text word by word so nothing is repeated, promote the last partial when the server never commits, hold short fragments until the sentence is complete, and reopen silent sessions after 8 s.

3. The 10-minute Live session limit. We open the next session before the old one ends and switch at a pause, then drain and de-duplicate: 0 lost and 0 duplicated sentences across forced rotations.

4. The obvious shortcut didn't work. We probed Gemini Live Translate for captions. Its text follows the synthesized speech and lagged 5 → 23 s, so it is great for voice interpretation but not for captions.

5. Billing surprises. We hit the free tier's 15 requests per minute, then a prepay balance of zero. Both are now documented as playbooks, so the next conference doesn't lose an hour to them.

What we learned

• Tail latency matters more than the median for live captions, and hedging is the cheapest fix when requests are short.
• How the client shapes the Live session (VAD, glossary, segmentation) matters more than any prompt.
• Each language view should show only its own language. The audience values consistency over completeness.
• Measure, don't promise.

What's next

Priority inference for guaranteed latency, the interface itself in English, Spanish and Portuguese (design brief ready), spoken interpretation with Gemini Live Translate for large rooms, and browser-microphone ingest for small events.

Built With

• asyncio
• caddy
• docker
• fastapi
• ffmpeg
• gemini-3.5-flash-lite
• gemini-3.5-transcribe-live
• gemini-3.8-flash-tts
• gemini-api
• gemini-live-api
• github-actions
• google-genai
• obs-studio
• pydantic
• python
• react
• shadcn-ui
• srt
• tailwindcss
• trivy
• typescript
• uvicorn
• vite
• websockets