LinguaStage

LinguaStage - Real-Time Multi-Stage Conference Captioning & Simultaneous Translation

Backend & Pipeline (Python / FastAPI): Asynchronous WebSockets: Handles low-latency bidirectional streaming between audio providers (microphones or live streams) and audience viewers across multiple concurrent stages. Local STT Engine (faster-whisper): Leverages CTranslate2-accelerated Whisper models to perform real-time speech recognition on CPU/GPU, emitting progressive partial word tokens followed by final sentence commitments. Dynamic Translation Engine (Gemini 1.5 Flash API + Fallback): Powered by Gemini 1.5 Flash API with custom prompt engineering to preserve developer jargon. Features a resilient fallback chain with in-memory LRU caching to eliminate rate limits. Technical Glossary Engine: Implements fuzzy phonetic correction to replace common speech-to-text misspellings (e.g., "cooper natives" -> Kubernetes, "postgress" -> PostgreSQL) before broadcasting to viewers.

Frontend & UX (Next.js / TypeScript / Tailwind CSS): Web Audio API: Captures 16kHz PCM audio directly from the speaker's microphone and streams binary frames over WebSockets. Live Subtitle Viewer: Supports Dual-View (Original + Translated), Spanish-only, and English-only modes with auto-scroll and live partial transcript indicators. Operator Control Panel: Real-time dashboard for event organizers to create stages, manage technical glossaries per session, and monitor stage status and active audience metrics.

Challenges I Faced: Low-Latency Streaming without CPU Starvation: Running multi-stage Speech-to-Text concurrently while keeping end-to-end latency below 500ms required optimizing buffer sizes, thread counts, and streaming partial word tokens prior to sentence finalization. Handling Technical Jargon: Standard translation models frequently misinterpret domain-specific terms. We solved this by inserting a pre-and-post translation Glossary Engine combined with context-aware Gemini prompts. Eliminating Rate Limits: Public translation APIs often impose strict rate limits. We engineered an LRU-cached dynamic RPC translation fallback chain to guarantee 100% dynamic subtitling at zero cost when an API key is missing.

What I Learned: Real-time Web Audio Pipelines: How to resample and stream raw PCM audio buffers efficiently over WebSockets from client browsers. Prompt Engineering for Translation: Structuring LLM prompts with glossaries to maintain technical fidelity across English, Spanish, and Portuguese. Stateless Multi-Stage Scaling: Designing async pipeline tasks so that a stage failure never cascades to other live conference streams.