Omnistage

Inspiration

At multi-track tech conferences and developer summits, attendees often struggle to follow live presentations due to language barriers, fast speech, and heavy technical jargon. Standard speech-to-text tools fail in these environments: they translate "Spring Boot" literally, confuse Kubernetes terms, and break down completely when speakers mix English and Spanish ("deployar en el cluster", "hacer un merge"). At the same time, live conference venues are notorious for unstable Wi-Fi. If an auditorium's captioning depends entirely on the cloud, a momentary network drop leaves the main projector blank. We created OmniStage to solve this: an enterprise-grade, zero-downtime, hybrid AI platform delivering real-time bilingual subtitling across multiple stages, seamlessly switching between Google Gemini in the cloud and local Gemma 4 on the edge whenever the connection drops.

What it does

OmniStage ingests live audio from up to 10 concurrent stages and delivers real-time synchronized Spanish and English subtitles across multiple surfaces. The system operates on an intelligent hybrid engine: under normal conditions, it streams continuous audio to Gemini Live (gemini-3.5-transcribe-live) and Gemini 3.5 Flash Lite for ultra-low-latency transcription and contextual translation with technical glossary protection. If internet connectivity fails, OmniStage instantly and automatically falls back to an offline pipeline running faster-whisper and a quantized Gemma 4 E2B model on the local GPU. Subtitles are distributed with sub-second latency to an attendee mobile PWA, an OBS Studio transparent broadcast overlay, a clean auditorium projector view, and a live operator monitoring dashboard, with automated exports to timestamped SRT, VTT, and TXT files as soon as each talk concludes. A custom anti-flicker reconciliation engine renders streaming draft hypotheses in soft gray that solidify cleanly into white committed sentences without layout jumps or visual distraction.

How we built it

We built OmniStage using a modular architecture that bridges real-time audio engineering, generative AI, and responsive frontend surfaces. The audio pipeline ingests RTMP streams through an embedded MediaMTX server, normalizes audio with FFmpeg (highpass filtering and loudness normalization to 16 kHz mono PCM), and applies Silero VAD to detect natural linguistic clause boundaries rather than arbitrary time slices. For the cloud pipeline, we implemented continuous streaming with the google-genai SDK, pairing Gemini Live for speech recognition with structured Gemini 3.5 Flash Lite calls for translation with injected glossary preservation. For the edge fallback, we integrated faster-whisper on CUDA alongside quantized Gemma 4 E2B (Q4_0) served locally via llama-server on consumer RTX hardware. The frontend is built in React and Vite, communicating over partitioned WebSockets with stateful snapshot replay so late-joining attendees immediately receive recent context. The entire solution is packaged as a standalone Windows desktop application using Electron and NSIS, bundling frozen Python runtimes, media binaries, and cuDNN libraries without requiring Docker or Redis, while protecting API credentials through Windows DPAPI via Electron's safeStorage.

Challenges we ran into

Balancing real-time latency with semantic completeness was our first major challenge. Slicing audio at fixed intervals either created cumulative delays or truncated mid-sentence thoughts, which degraded translation quality. We solved this by using Silero VAD to emit overlapping 1.5-second drafts anchored to stable clause start timestamps, committing only when a natural pause occurred. Another major hurdle was handling bilingual developer jargon and Spanglish, as standard models compulsively translate command names and technical terms. We resolved this through dynamic agenda glossary injection and strict system prompts in both Gemini and Gemma 4. Achieving seamless, glitch-free failover between cloud WebSockets and local CUDA inference without repeating or dropping words required precise monotonic watermark tracking and replay buffers. Finally, bundling Python, CUDA drivers, MediaMTX, and Node into a single, reliable native Windows installer required extensive environment sandboxing and custom NSIS scripting.

Accomplishments that we're proud of

We are proud to have built a true zero-downtime hybrid pipeline where you can physically disconnect the internet mid-talk and watch the local Gemma 4 and faster-whisper engine seamlessly take over subtitle generation within seconds. We successfully ran Gemma 4 E2B on a consumer laptop GPU with under 1.8 GB of VRAM, achieving responsive, jargon-aware translations entirely on-device. We also designed a broadcast-grade reconciliation UX that completely eliminates subtitle flicker, delivering a reading experience on par with professional television captioning. Finally, we backed the entire platform with an extensive automated test suite covering over 96 backend tests, Playwright browser flows, and simulated multi-stage live loads.

What we learned

We learned that acoustic-aware segmentation based on natural breathing pauses vastly outperforms fixed time slicing when feeding audio to language models for translation. We also discovered how remarkably capable Gemma 4 E2B is at specialized, domain-specific tasks when guided by focused system instructions and constrained outputs, even under 4-bit quantization. Most importantly, we learned that designing for physical live events demands prioritizing operational resilience above all else: having an immediate, high-quality on-device edge model is what transforms an AI demo into a dependable production tool.

What's next for Omnistage

Next for OmniStage is completing the QLoRA fine-tuning of Gemma 4 on our ethically gathered, authorized conference recordings to bake regional tech terminology directly into the model's weights. We plan to introduce live acoustic speaker diarization to automatically identify and color-code panelists during debate sessions. We also intend to expand our language offerings to Portuguese, French, and Japanese for international tech summits. Lastly, we will connect our public Vercel frontend with Cloudflare Tunnels, allowing thousands of arena attendees to stream live captions directly on their mobile phones over 5G without burdening local venue Wi-Fi networks.