Open Live Caption

Open-source real-time transcription & translation for conferences, powered by Gemini's Live API scales to 10+ simultaneous sessions, with live mic input and multi-language captions.

Comment

Inspiration

Nerdearla runs 30+ talks per event and needs live captioning to stay accessible, but today that's solved with expensive commercial tools plus manual operation. It doesn't scale to a multi-track conference. We wanted an open source alternative that any conference, not just Nerdearla, could deploy without depending on paid vendors or a person babysitting each session.

What it does

  • Transcribes live audio and translates it in real time (tested with English, Spanish and Portuguese).
  • Takes audio from a test file or a real microphone captured directly in the browser. Genuinely live audio, no pre-recorded files required.
  • Runs multiple isolated sessions ("stages") in parallel. Validated with 10 concurrent sessions, no degradation.
  • Audience-facing web view: pick a stage and a language, see live closed captions with a typewriter animation.
  • Auto-corrects technical terms and proper nouns via a lightweight glossary, with zero added latency.
  • Production monitoring panel (/admin): live status per session, connected clients, and a "close session" control to kill a session on demand.
  • Exports the full transcript per talk (SRT / VTT / plain text).
  • Keeps the translation pipeline in sync with pause/seek/resume on the video.

How we built it

FastAPI and native WebSockets on the backend, talking to Gemini's Live API (gemini-3.5-live-translate-preview) through the google-genai SDK. Each (stage, language) pair is a fully isolated Session: its own Live API connection, its own buffers, no shared state with any other session. That isolation is what makes scaling trivial. Going from 2 to 10 sessions is just opening more connections, not touching the architecture.

For live audio we skipped native audio libraries entirely and capture microphone input directly in the browser (getUserMedia plus AudioContext and ScriptProcessorNode), streaming raw PCM16 over a binary WebSocket to the backend. Transcripts persist to SQLite for export. The frontend is plain HTML/CSS/JS, no framework, with a custom dropdown/settings UI built to match a real streaming player's UX.

Challenges we ran into

  • No native audio on Windows: pyaudio had no wheel for our Python version and needed build tools we didn't want to force on every deployer. We moved microphone capture to the browser instead, which turned out simpler and removes a whole class of platform-specific install pain.
  • Video/caption drift: pausing or seeking the demo video would desync it from the translation pipeline. We built a real sync protocol (generation counters and control messages over the WebSocket) instead of faking it. A late-joining viewer or a seek now correctly restarts the pipeline at the right offset.
  • A subtle Gemini Live API flag: with echo_target_language=False, the API silently stops emitting translated output when the target language matches the language actually being spoken. Captions just went blank with no error. Tracing that down to one boolean was the trickiest bug of the build.
  • Cost and quota management as a first-class concern: background sessions keep looping by design, so a late viewer never finds a dead session, but that means idle sessions quietly keep burning API quota. We hit our own spending cap mid-build from forgotten load-test sessions, which pushed us to add a manual "close session" action to the monitoring panel instead of relying on restarting the whole server.
  • Cutting a feature we couldn't do right: we prototyped live camera video (periodic JPEG snapshots), but it never looked like real live video without WebRTC. We removed it rather than ship something that looked broken, and documented it as a roadmap item instead.

Accomplishments that we're proud of

Real, measured numbers instead of guesses: 10/10 concurrent sessions succeed with about 2.2s average latency to first caption. Genuinely live microphone input with zero native audio dependencies. A sync system robust enough that pausing, seeking, or a late-joining viewer never breaks the caption stream.

What we learned

Building for a live event forces you to treat API cost and quota as part of the architecture, not an afterthought. A background loop that's great for reliability (sessions never "die" on a late viewer) is also a cost risk if you don't give operators a way to shut it down. We also learned to read translation API config flags very literally: a default we assumed was harmless (echo_target_language=False) silently broke a whole use case.

What's next

OBS/vMix integration to burn captions directly into a stream, a fully local mode using Gemma for teams that can't depend on an external API, an audio "listen mode" using the translated audio the Live API already generates internally, and broader language coverage tested end to end.

Built With

Share this project:

Updates

Submission history