🌟 1. What Inspired Us: The Hackathon Spark & The Mission At international developer conferences like Nerdearla—one of Latin America’s largest tech gatherings—dozens of stages run simultaneously. World-class engineers take the mic to talk about distributed systems, cloud-native architecture, and the future of AI. While many keynotes are delivered in English, the vibrant local and remote community across Latin America is predominantly Spanish-speaking (along with Portuguese-speaking peers across Brazil and LATAM).
Too often, attendees face an invisible barrier:
Human simultaneous interpreters are fantastic, but booking and coordinating interpreters across 5, 10, or 30 parallel stages is logistically and financially out of reach for community-driven events. Traditional speech-to-text pipelines (batching audio into 5–10 second chunks, running Whisper, and sending text to a translation API) introduce 4 to 8 seconds of latency. By the time the subtitle appears on screen, the speaker has already changed slides and moved on to the punchline. We asked ourselves: What if we could bring subtitle latency down to sub-second, token-by-token streaming, making live talks seamlessly accessible on attendee phones, production monitors, and live OBS streams?
When the Google Gemini Live API introduced bidirectional, ultra-low-latency audio streaming (gemini-3.5-transcribe-live and gemini-3.5-live-translate-preview), we knew this was our moment to build Nerdearla Live Subs.
🧗 2. The Challenges We Faced (And How We Solved Them) Building for live audio under hackathon pressure pushed us to solve several non-trivial engineering problems:
Challenge 1: Capturing Audio from Live YouTube Video Tabs The Problem: Web security and the Same-Origin Policy (CORS) strictly prohibit JavaScript from intercepting audio directly from cross-origin embeds. The Solution: We implemented native browser tab capture via navigator.mediaDevices.getDisplayMedia({ audio: true, video: true }). By instructing users to select the Chrome Tab with the talk and checking "Also share tab audio", the browser hands over clean, uncompressed digital audio directly from the video player. Challenge 2: Eliminating UI Thread Audio Glitches The Problem: Initial attempts using the legacy ScriptProcessorNode ran on the browser’s main UI thread. Whenever the DOM updated with rapid new subtitle tokens, the audio thread dropped frames, corrupting the PCM stream sent to Gemini. The Solution: We migrated to modern AudioWorkletNode, executing the downsampling and Int16 conversion on a dedicated real-time audio thread completely decoupled from the DOM. Challenge 3: Bluetooth Microphones & Hardware Clipping The Problem: Speakers using Galaxy Buds or Bluetooth headsets frequently suffered from severe voice clipping because standard browser audio filters (echoCancellation, noiseSuppression) aggressively stripped high-frequency phonemes. The Solution: We added hardware configuration toggles that disable browser software clipping, paired with a custom in-browser Mic Boost slider (+3.5 dB gain) to keep the speech at the optimal input volume for Gemini's acoustic models. Challenge 4: Technical Jargon, Lunfardo & Spanglish The Problem: Generic translation models often awkwardly translate developer jargon (e.g., translating “thread pool” to “pileta de hilos” or “Kubernetes cluster” to “racimo de timoneles”), and stumble over Argentine voseo (“mirá”, “fijate”). The Solution: We crafted specialized system prompts and glossary instructions ensuring technical terminology (Kubernetes, Kafka, gRPC, CI/CD, asyncio, PostgreSQL) remains in standard developer English while naturally embracing regional Latin American Spanish.</p> <p>💡 3. What We Learned Streaming AI Demands a New Architectural Mindset: Moving from request/response LLM calls to bidirectional, persistent streaming requires thinking deeply about FIFO queues, backpressure, and handling speculative interim tokens that mutate as the speaker continues speaking. The Synergy of Go and Web Audio: Pairing browser-side audio worklets with a concurrent, low-overhead Go backend is an unbeatable combination for real-time media processing. It allowed a single cheap instance to handle multiple stages effortlessly. Accessibility Unlocks True Community: Lowering latency from 6 seconds to under 1 second fundamentally changes the experience. It transforms subtitles from a distracting after-the-fact transcript into an invisible, natural bridge that lets everyone in the room react to the speaker's jokes and insights at the exact same moment. Nerdearla Live Subs was born out of love for the tech community, open source, and the belief that language should never be a barrier to sharing knowledge.</p>
Built With
- and-screen/tab-capture-api-(navigator.mediadevices.getdisplaymedia)-frontend-&-ui:-vanilla-html5
- audioworkletprocessor-(16khz-pcm-downsampling)
- channels
- coder/websocket)-&-python-3.11-/-fastapi-(initial-prototype)-ai-&-speech-models:-google-gemini-live-api-(gemini-3.5-transcribe-live-&-gemini-3.5-live-translate-preview)-via-bidirectional-websockets-audio-&-media-ingestion:-web-audio-api
- css3
- gemini
- golang
- html5
- javascript
- javascript-(zero-external-dependencies/frameworks)-broadcasting-&-output:-websockets
- obs-studio-/-vmix-transparent-browser-overlays
- server-sent-events-(sse)
- sse
- webaudioapi
- websockets
Log in or sign up for Devpost to join the conversation.