Subtitula

Open source live captions and simultaneous interpretation for every room of a conference, on the audience's phones, with the Gemini Live API. 0.3 s latency, US$ 2.21 per room-hour.

Comment

Try it: open https://flordelcastillo.github.io/subtitula/ on your phone, tap "Play the talk", switch captions between Español, English and Português, and tap "Listen" with headphones on. It replays a real run of the live engine; the interface follows your phone's language. Video: https://youtu.be/59H_dzXcG7s

Inspiration

Nerdearla captions every talk today with two commercial tools, one for Spanish transcription and one for English to Spanish live translation, operated by hand. This year there are more than 30 English sessions, many in parallel. We wanted one open source tool that any conference can deploy, that costs little, and that the production team can run from a single screen.

What it does

  • Every room, live. Each stage streams its audio (browser tab, microphone, HLS, RTMP, SRT or YouTube) into a Gemini Live API session per target language (gemini-3.5-live-translate-preview). The original transcript and the translation arrive word by word while the speaker talks.
  • For the audience. People scan a QR code and read the talk on their phone in Spanish, English, Portuguese or the original. They can listen to the spoken interpretation with headphones: Live Translate already generates that voice, so it costs nothing extra. A "What did I miss?" button summarizes the last 5 minutes in their language with Gemma 4.
  • For production. A room screen with big captions and a QR code, an OBS/vMix overlay (transparent or chroma), a live.txt feed for vMix titles, and a dashboard with audio level, latency p50/p90, errors, live sessions, real cost per room and a glossary you can edit live.
  • Captions done right. Each language is its own track, split into sentences of at most 84 characters (the 2 x 42 subtitling standard), so Spanish reads like Spanish, not like English cut in the wrong places.
  • Cheap at scale. Secondary languages only open a Live session while someone reads or listens to them, and long silences are not streamed. The cost comes from official prices: US$ 2.21 per room-hour per language, about US$ 44 for 30 talks of 40 minutes.
  • Never lagging. Organizers told us lag is the worst experience: they would rather have captions cut and resume than run 10 s behind. A watchdog checks every session twice per second and, if a translation stalls or falls more than 6 s behind, reopens it live without carrying the backlog.
  • Operable by a small team. Per-room agenda: each talk's title, speaker, language and glossary switch by themselves at their scheduled time. AI-suggested glossary per talk, Prometheus /metrics, and a single /admin for every room. Any room can also be fed from a browser tab, with no install.
  • Three engines. Live (default), chunked (gemini-3.5-transcribe plus Flash-Lite or Gemma translation with failover), and fully local (faster-whisper plus Gemma 4 via Ollama).

How we built it

Python and FastAPI. One worker per room, with no shared state, and a hub that only moves text: Server-Sent Events with Last-Event-ID so phones never lose a line on reconnect, plus a WebSocket for the interpretation audio. Plain HTML, CSS and JS with no build step, designed for dark rooms and phones (Atkinson Hyperlegible, high contrast mode, 44 px touch targets, interface in Spanish, English and Portuguese). Docker Compose runs one container per room. 34 tests run in CI without an API key.

Challenges we ran into

  • Our first design sent each 3 to 5 s chunk to a text model. On hackathon day the models were saturated (503) and the free tier allowed only a few requests per minute, so we moved to the Live API, where one open session per room avoids per-request limits.
  • Aligning the translation line by line with the original turned out to be impossible live, because the translation arrives with a variable lag. We split each language into its own caption track.
  • Our dashboard first underestimated the cost about 18 times (text prices instead of audio prices). We fixed it with the official price table and verified it against measured usage.

Accomplishments that we're proud of

  • Latency with evidence: over 10 continuous minutes of each sample talk, the translation arrives 0.29 to 0.34 s (p50) after the speaker pauses, under 0.9 s p90, with no accumulated drift.
  • Scale test: 20 rooms and 500 viewers with word-by-word updates: 4,223 deliveries per second, 4 ms p50 from hub to screen, 0 errors. Also tested with 50 rooms and 1,000 viewers. Scripts and raw results are in docs/evidencia.
  • Spoken interpretation and "What did I miss?" on the audience's phone, and the captions of our own demo video made with Subtitula.

What we learned

The Live API changes the economics of live captioning: one session per room and language, billed per minute, gives both the transcript and the translation. Building on a preview model means designing for failure: session resumption, context window compression, reconnections that never take other rooms down, and fallback engines.

What's next for Subtitula

Import the official conference schedule directly instead of writing it in sessions.yaml, publish a container image, and a Gemma 4 native audio engine for fully offline events.

En español

Probalo: abrí https://flordelcastillo.github.io/subtitula/ en el celular, tocá "Reproducir la charla", cambiá los subtítulos entre español, inglés y portugués, y tocá "Escuchar" con auriculares. Repite una corrida real del motor en vivo.

Subtitula da subtítulos e interpretación simultánea en vivo para todas las salas de una conferencia, en el celular del público, con la Gemini Live API. Cada sala manda su audio (navegador, micrófono, HLS, RTMP, SRT o YouTube) y el público escanea un QR para leer la charla en español, inglés o portugués, unas 0,3 s después de cada pausa del orador, escuchar la interpretación con auriculares o pedir "¿Qué me perdí?" para un resumen de los últimos 5 minutos con Gemma 4. Producción tiene pantalla de sala con QR, overlay para OBS/vMix, agenda por sala, glosario editable en vivo y un panel con demora, errores y costo real: US$ 2,21 por sala-hora, unos US$ 44 para 30 charlas. Un vigía reabre en vivo cualquier sesión que se atrase más de 6 s, porque la organización nos dijo que lo peor es el atraso. Open source con licencia Apache 2.0, 34 tests en CI, Docker Compose y un motor 100% local con faster-whisper y Gemma.

Built With

Share this project:

Updates

Submission history