Inspiration
Open-source tech conferences like Nerdearla thrive on accessibility and community inclusion. However, providing real-time transcription and simultaneous translation for 30+ parallel sessions quickly hits a wall with commercial SaaS solutions — they are prohibitively expensive, closed-source, and heavily dependent on manual human oversight per room.
My motivation was simple: build a lightweight, zero-operator open source alternative capable of running multi-room live captioning seamlessly. As a solo developer, I wanted to prove that a clean, event-driven architecture powered by modern AI models could democratize live event accessibility for open-source communities worldwide.
How I Built It
StreamScript AI is a single FastAPI process that runs one asyncio task per active room — no message broker, no per-room container, no microservices. Each task owns a WebSocket ingest endpoint, a transcription provider, and a pub/sub hub that fans out captions to every viewer watching that room.
The core pipeline is two-stage by necessity, not by choice: I first tried to get a single Gemini Live session to accept audio in and return transcription + translation as text in one round trip, but no current Gemini conversational Live model supports that combination — audio-in sessions are built for audio-out. So the real architecture streams audio to gemini-3.5-transcribe-live for low-latency ASR, diffs the cumulative interim transcript on sentence boundaries, and fires a separate generate_content call (JSON-mode, with retry/backoff) to translate each finished sentence. If translation fails after retries, the caption still ships in the source language rather than being dropped — a degraded flag tells the frontend to show it was un-translated instead of silently hiding it.
For the fully offline path, I built a second provider behind the same TranscriptionProvider interface: faster-whisper (CTranslate2, CPU-only) for ASR plus a local Gemma model served through Ollama for translation, with RMS-based silence-gap segmentation to decide when an utterance is "done." Swapping providers is a single environment variable — the rest of the system (rooms, WebSocket hub, export, dashboard) doesn't know which one is running.
On top of that: a technical glossary gets injected into every translation prompt so domain-specific framework and library names don't get mistranslated; an audience view with a QR code and a language picker; an OBS/vMix-ready overlay; SRT/VTT/TXT export with timestamps rebased to be relative to each room instead of absolute epoch time; and a production dashboard for monitoring multiple live rooms at once. Room creation and closing are gated behind an admin token (constant-time comparison) so the public deployment isn't open-write by default. The whole thing ships as one docker compose up, with 51 automated tests and CI running on every push.
Challenges I Faced
Concurrency vs. latency. Managing real-time audio for multiple simultaneous rooms as a solo developer meant leaning entirely on asyncio — one task per room, no shared blocking state — to keep a burst of traffic in one session from adding latency to captions in another. I validated this directly with up to 8 concurrent rooms against the real Gemini API before trusting the design.
The API didn't work the way I originally assumed. I planned around a single Gemini Live session doing audio-in/text-out transcription plus translation in one shot. Real testing showed that's not how the current conversational Live models behave, which forced the two-stage ASR + translate architecture described above — a mid-build redesign, not a footnote.
Technical jargon accuracy. Standard speech-to-text models frequently mishear specialized developer framework and library names, and a generic translation call will happily "correct" jargon into nonsense. Injecting a dynamic domain glossary into the system prompt was essential to stop translation hallucinations on exactly the vocabulary a tech conference audience cares about most.
Building solo, under real deadline pressure. Balancing pipeline correctness, UI/UX for attendees, security (auth on room mutation endpoints), test coverage, and documentation required strict scope discipline — and, in the final stretch, working around a local disk-space failure on my own dev machine while pulling a large local-model image, without letting it block shipping the rest of the submission.
What I Learned
Audio-native models still need a two-stage shape for structured output — today. I expected a multimodal model to collapse transcription + translation into a single call. In practice, current Gemini Live conversational models are tuned for audio-in/audio-out, not audio-in/text-out, so getting reliable structured captions still meant streaming ASR plus a discrete translation step. That's a real, current API constraint worth knowing before you architect around the "one model does everything" assumption.
Architecture over complexity. A well-designed, event-driven asyncio pipeline handled high-concurrency multi-room workloads without needing a broker, a queue service, or a cluster — the interface boundary (TranscriptionProvider) mattered far more than the infrastructure around it.
The power of open source. Designing a tool meant for other conferences to self-host forces cleaner, more modular code — auth, environment-based config, and a single docker compose up aren't nice-to-haves when the deploy target is "anyone."
Log in or sign up for Devpost to join the conversation.