Vox Libera

Open source live captions and translation for every stage of your conference, powered by Gemini Live. Tested: 10 stages, 0 errors, ~1 s.

Comment

Inspiration

Nerdearla has offered live Spanish captions for years, using two commercial tools: one for Spanish→Spanish transcription and another for English→Spanish live translation. This year there are 30+ English sessions, many running at the same time, and that setup doesn't scale: it's expensive, it needs manual operation for every stage, and no other community conference can replicate it.

We wanted the opposite: an open source system that any conference can deploy in minutes, where adding a stage is one line of config.

What it does

Vox Libera takes the live audio of each stage and delivers real-time captions in the original language plus Spanish, English and Portuguese:

  • 📱 Audience view: people pick a stage and a language on their phone (shareable link / QR).
  • 🎙 Broadcast from the browser: the stage crew sends the mixer's audio (microphone, a browser tab or a file). Nothing to install.
  • 📺 OBS / vMix overlay: one URL as a browser source, transparent background, captions burned into the stream.
  • 📊 Production dashboard: status of every stage, translation latency, errors, the last sentence heard and the live cost per stage, with one button for every action.
  • ⬇ Exports: the full transcript as SRT / VTT / TXT, in any language.
  • 📖 Glossary per stage (speaker names, jargon), used both for speech recognition and for translation.

The English subtitles of our demo video were generated live by Vox Libera itself.

How we built it

  • Backend: Python + FastAPI, fully async. One pipeline per stage, with no state shared between stages.
  • Transcription: gemini-3.5-transcribe-live over the Live API, streaming 16 kHz PCM in 100 ms chunks.
  • Translation: gemini-3.5-flash-lite. One request per sentence returns every language at once, so adding a language costs no extra requests.
  • Frontend: plain HTML + JavaScript, no build step. It works offline on a venue network, with an EN/ES UI. The browser captures audio with an AudioWorklet that resamples to 16 kHz.
  • Ops: docker compose up, a .env file and an admin key. There's also a CLI for unattended sources (RTMP/SRT/HLS streams, YouTube, audio interfaces).

We measured before designing. We ran the Live API against real Nerdearla talks, and every design decision in the README comes with the experiment behind it.

Challenges we ran into

  1. Speakers don't pause. Gemini only finalizes an utterance on a pause, and fluent speakers went 50+ seconds without one. Waiting for finals would have made translations useless. Instead, we translate each sentence as soon as it is complete and stable in the live (interim) transcript, and use finals only to correct what's already on screen.
  2. Sessions expire. A Live session lasts ~590 s. The server sends a GoAway 50 s before the end and aborts with error 1008 if the client doesn't close. We built seamless rotation: open a second session, feed both, hand over at the next sentence boundary and strip the duplicated words. Talks of any length now work with no lost or repeated words.
  3. The model rewrites itself. Interim text gets revised in place and sometimes glues sentences without a space ("World War II.the"). We handle both with a dedicated segmenter, covered by tests built from real transcripts.
  4. "Smart" mode was too smart. It turned speech into Markdown bullet lists and dropped content. We went with verbatim transcription and let the translation prompt remove filler words, with explicit instructions never to summarize.
  5. Bugs you only see live. Recording the demo exposed a stopped browser tab that kept sending audio, which then got mixed into the next broadcast. We fixed it with per-source generations and a regression test. We found several issues like this by testing with real audio, not mocks.

Accomplishments we're proud of

Load test on a single instance: 10 stages transcribing in parallel + 200 viewers → 0 errors, 91,400 caption messages delivered in 3 minutes, ~150 MB of RAM and ~8% of one CPU core.

Metric Value
Original text on screen < 1 s after it's spoken
Translation latency ~1 s
Cost ≈ US$ 0.77 per stage-hour (3 languages)

The cost is linear in stages: for an event with $S$ stages running $H$ hours,

$$\text{cost} \approx 0.77 \cdot S \cdot H \ \text{USD}$$

so 10 stages over a 2-day, 8-hour-per-day event cost about US$ 123.

What we learned

  • Measure the API before designing around it. Our first plan (sending 3–5 s audio chunks) would have been slower and cut words at every chunk border.
  • Latency is a pipeline property. Once the model was fast, the real delays came from when we decided a sentence was done.
  • Operations are a feature. Non-technical crews need buttons, clear states and "is it hearing the speaker?" at a glance, not logs.

What's next

Translated audio for headphones (Gemini Live Translate), glossary editing from the dashboard and loaded from the event agenda, QR codes per stage, a Redis broadcaster for multi-replica deployments, a fully local mode with Gemma for events with no budget or no internet, and committing long sentences at clause boundaries. Right now fast, run-on Spanish speech gets its translated caption ~4 s after the sentence ends, versus ~1 s in English.

Built With

Share this project:

Updates

Submission history