OpenCaptions

Inspiration

At Nerdearla, talks run in parallel in Spanish and English. If you don't speak the talk's language, you're hard of hearing, you watch the stream, or you simply walked in ten minutes late, you miss the content. The crew already routes each sound desk through a 3.5 mm cable into a mini PC with a browser. We wanted something that fits that exact setup, is open source, and helps people — not just prints subtitles.

What it does

Live captions + translation on your phone. Scan the room's QR, pick Original / Español / English / Português. Big readable text, reading settings (high-legibility fonts, size, spacing, light/dark), and optional translated voice in your headphones.
✨ What did I miss? A summary of the last 5 minutes (or the whole talk) in your language, generated from the transcript.
💬 Ask the talk. "Which tool did she use for tracing?" — answered only from what was said, with quotes and timestamps; if it wasn't said, it tells you.
Transcripts for everyone. A live transcript page and a library of every talk: search, switch language, download TXT/SRT/VTT.
Bilingual speakers. Hosts switch between Spanish and English all the time: every caption language follows the speaker (transcription when they speak it, translation when they don't).
For the stage and the stream. A projector view with big captions and a QR, and a transparent overlay for vMix/OBS (optionally bilingual, two languages at once) so remote viewers get translation too. OBS/vMix can stream straight to a room over RTMP. On a laptop, captions can float on top of any window.
For organizers. npm run setup in one minute, a dashboard with a getting-started checklist, latency, cost and alerts per room, printable QR posters, a floating mini-dashboard that stays on top of OBS, and an agenda pasted straight from a Swapcard export that names every talk automatically and teaches the AI the speakers' names. Rooms run themselves: silence gating (nothing billed between talks), automatic resume, reconnection, and talk splitting.

How we built it

Node.js server, a single process with no build step and five dependencies. Vanilla JS pages. MIT license.
Gemini 3.5 Live Translate (Live API, WebSocket) gives us one streaming session per room: the original transcription with the detected language, plus translated speech.
Gemini 3.5 Flash-Lite translates each sentence with context and a technical glossary. Provisional translations appear while the sentence is still being spoken. Flash-Lite also powers the grounded audience assistant: JSON output, cached and rate-limited, with a transcript-highlights fallback.
Audio in:
- a headless agent (ffmpeg → PCM 16 kHz → WebSocket) running as a system service on each room PC;
- or a browser AudioWorklet page;
- or a server-side pull of the SRT/RTMP/HLS stream vMix already produces.
Security and operations:
- Secure by default: auto-generated tokens and a localhost trust model.
- CSP and rate limits.
- SSRF-safe pulls.
- A hardened Docker image, a systemd unit and Cloudflare Tunnel support.
- Prometheus metrics.
- A full Diátaxis documentation set with ADRs, an event-day runbook in English and Spanish, and CI on Linux, macOS and Windows.

Challenges we ran into

Speech-to-speech translation drifted behind on long, fast talks. We measured it with real Nerdearla talks and switched captions to sentence-level text translation, with provisional updates. That bounded translation delay at about 4.6–5.7 s, against about 3.2 s for the original captions.
Bilingual hosts. Nerdearla hosts switch between Spanish and English. Every caption track now follows the speaker: it shows the transcription when the speaker uses that language and a translation when they don't. Hysteresis stops a single English word from flipping the captions.
Quotas. The free tier returns HTTP 429s. We built a global limiter, backoff, and an automatic fallback to Live Translate's own captions, so the screen never freezes.
Live-event failure modes. Our pre-launch review found crash vectors and races that only show up live: oversized frames, streams that aren't live yet, session renewal every ~10 minutes losing words, and talk rollovers mid-sentence. We fixed each one and added regression tests.

Accomplishments that I'm proud of

30 rooms at once, for real. We ran 30 Nerdearla 2025 talks through Gemini simultaneously on one laptop. Captions stayed at p50 1.2 s / p90 2.7 s (translation 1.8 s / 3.5 s) after speech onset, with the server at ~20 % of one CPU core, 151 MB of RAM and a 1.5 ms event-loop p99. The whole run cost US$ 2.38, about US$ 1.2 per minute for 30 rooms. Spanish, English and Portuguese talks were detected automatically.
Zero-operator rooms: nobody has to press anything during a talk.
An assistant that is honestly grounded. It quotes the transcript or says "not mentioned", in the viewer's language, and costs well under a cent per talk.

What we learned

Latency is roughly 80% model time. What moves the needle is the right model per job (Live for recognition, Flash-Lite for text), pinning the source language, and a clean desk feed. Faster hardware doesn't help.
For accessibility at events, reliability and ease of setup matter as much as model quality.

What's next for OpenCaptions

A fully local engine with Gemma for events without internet (the engine interface is ready).
Streaming translation to save another 0.2–0.5 s.
Speaker labels.
SSO for the dashboard.