Poliglota

• 

  Poliglota

Inspiration

Nerdearla runs talks in parallel all day, and a good chunk of the audience misses content because of language. Hiring simultaneous interpreters for every room doesn't scale, and generic captioning tools fail exactly where it matters most: a speaker says "the bottleneck is in the vector database" and it comes out transcribed as gibberish.

What it does

Políglota takes live audio from a talk and produces captions in the original language, translated on demand into whatever languages the audience is asking for. Multiple rooms run in parallel.

The overlay projected in the room carries a QR code: attendees scan it and get the captions on their own phone, in their own language, no install required. It's the difference between one screen forcing everyone into the same language and each person reading in theirs.

Since the last update, captions also know who's talking. Each speaker enrolls their voice once, and from then on lines show up as Juan: …, Pedro: …, and, for anyone unregistered, Unknown: …. It runs entirely local — no extra API calls, no added latency on the ASR path.

Four screens ship with it: an event operator panel, an audio-capture page for the sound booth, an audience view with a language picker, and a transparent OBS overlay.

How we built it

Backend in Python with FastAPI. Audio comes in over WebSocket as 16 kHz PCM16 from the browser, from ffmpeg, or from any client. It goes to gemini-3.5-transcribe-live in streaming mode; every closed sentence gets translated with gemini-3.5-flash-lite to every active language in a single call, with prior sentences as context and a glossary that keeps technical jargon from being localized. A pub/sub bus per room fans out to N viewers without a slow client stalling the rest.

Captions follow the roll-up pattern from live TV, with a rule almost nobody implements: minimum on-screen time proportional to line length (15 characters per second), so three back-to-back sentences from the speaker don't erase the first one before anyonetpaces what's readable, the system drops that minimum and prefers to catch up.

Speaker identification runs on top of the audio the room is already processing — no extra network round-trip. A speaker's voice is embedded into a vector (classic MFCC+pitch features, or a neural embedding via Resemblyzer when available), and each clip is matched against every registered speaker by raw distance to that speaker's centroid, gated by that speaker's own spread: a clip counts as "Juan" only if it's within $N$ times Juan's own historical varrything else is "Unknown." Engines for both ASR and translation are swappableby environment variable; the whole thing also runs 100% local with faster-whisper and Gemma over Ollama.

Challenges we ran into

We wrote the pipeline against the docs, and almost everything important turned out to be wrong.
interim_input_transcription is the live stream and it's cumulative, not incremental — we were reading the wrong field, so nothing showed up until the audio finished. The model also rewrites what it already transcribed, so identifying sentences by content made a correction show up as a new line and the same phrase print five times. The most instructive one was latency. With two rooms, the first caption took ike a provider quota limit. It was our own 256-chunk audio queue: if the session took time to establish, 25 seconds of stale audio piled up and the system started transcribing the past. Capping it at 5 seconds brought the startup down to **2.1 seconds. Another: the browser's AudioWorklet delivers 128-sample blocks — 8 millisecon one by one. That's 125 messages a second choking the API; first caption took 30 to 60 seconds. Batching to 100 ms dropped it to 2.8 seconds.

And one on operations: a room opened a Live session as soon as it was created, even with no audio. The server aborts a session that receives nothing with a 1008, and it takes about 60 seconds to do it — a room created ahead of the talk kept connecting and dying in a loop.

Speaker identification had its own sharp edge. The ranking rule picked the clostance} / \text{own spread}$, and a wider voiceprint divides by more — itsystematically won every tie. Measured with real microphone recordings via leave-one-out: 4 of 12 clips got attributed to the wrong person. Switching to ranking by raw distance (using spread only to gate whether the closest match counts as a match at all) fixed 1

Accomplishments that we're proud of

That every number in the README is measured against the production API and reproducible from the repo's own tests. And having pulled a made-up "end-to-end latency" metric out of the panel: the model accepts word_timestamp but never returns per-word timing, so there'ord with the instant of audio that produced it. Any live "round-trip delay"shown would have been a lie, so the panel now reports only refresh rate and time-to-first-token — what's actually measurable.

What we learned

                                                                                                                                                                                        That the glossary didn't do what we thought it did. The premise was priming theargon via `adaptation_phrases`. Measured: the minimal configuration nails 10 of10 technical terms, and vocabulary biasing didn't improve accuracy while tripliit to the translator instead, where it actually earns its keep — keeping"garbage collector" from turning into a Spanish phrase about trash collection in the middle of a JVM talk.                                                                           

For speaker ID, we learned that a naive similarity threshold isn't enough once voices are recorded on a real microphone instead of synthetic TTS: two YouTube speakers with a shared recording setup sat close enough in embedding space that the wrong ranking ruley. Self-calibrated, per-speaker spread — not a single global threshold — is what actually separates people whose voices happen to be similar.

What's next for Políglota

Validating, against the real API, the voice-activity gate that currently ships disabled: it holds back silence from a room open with nobody talking, but we didn't have the time to confirm it never clips a word, and accuracy is worth more than the tokens it would saveation once a talk closes (live diarization measured at 8× the caption refreshcost), and exporting WebVTT so a recording comes out captioned without a second

Built With

• accessibility
• apache-2.0
• asyncio
• docker
• fastapi
• faster-whisper
• ffmpeg
• gemini
• gemini-live-api
• gemma
• google-genai
• javascript
• obs
• ollama
• python
• real-time
• resemblyzer
• speaker-identification
• speech-to-text
• translation
• uvicorn
• voice-embeddings
• webaudio
• websockets