## What is CaptionMesh?
CaptionMesh is an open-source real-time captioning and translation system built during the **Nerdearla Vibeathon 2026**.
It receives audio from a configured source, sends it to Gemini for real-time speech transcription, translates the resulting captions from English to Spanish, and delivers them to an audience through a web interface.
The current implementation uses a modular asynchronous architecture. Each stage has its own audio source, transcription session, translation tasks, event stream, and audience connections.
## What inspired it?
The initial idea came from live technical talks where the content is useful to an international audience, but language can make it harder to follow.
We decided to explore whether a relatively small open-source system could provide live transcription and translation without introducing a large distributed architecture.
Rather than treating captions as a post-processing task, we built the system around a streaming pipeline:
```text
Audio
↓
Real-time transcription
↓
Caption events
↓
Asynchronous translation
↓
WebSocket
↓
Audience
This also gave us a practical way to experiment with the latency and reliability characteristics of Gemini's real-time API.
How we built it
The application is implemented in Python using FastAPI and asyncio.
The transcription layer uses Gemini Live, while translation uses Gemini Flash-Lite. FFmpeg is used for audio input and normalization, and WebSockets are used to deliver caption events to the audience.
Caption events are stored as JSONL. From those events, the application can generate WebVTT files for completed sessions.
The system also supports multiple independent stages. A stage can have its own audio source and Gemini sessions while sharing the application infrastructure.
What we learned
The main challenge was not getting a transcription working, but making the streaming behavior reliable.
Gemini's real-time transcription produces interim and final results, so we had to decide when an interim result represented a complete caption and how to avoid publishing the same text again when the final result arrived.
We also had to handle long-running sessions, session restarts, asynchronous translation, and cancellation of network operations that could remain blocked longer than expected.
Another important lesson was keeping the architecture simple. Instead of introducing separate services for each component, we kept the system as a modular monolith with independent asynchronous pipelines.
Challenges
The main technical challenges were:
- Producing useful caption-sized segments from streaming transcription results.
- Avoiding duplicate captions between interim and final transcription events.
- Keeping translation independent from the transcription stream.
- Handling Gemini session lifecycle and reconnection.
- Generating a separate WebVTT artifact for each completed session.
- Supporting independent stages without introducing unnecessary infrastructure.
What we built during the Vibeathon
The resulting MVP includes:
- Real-time audio input.
- English speech transcription.
- English → Spanish translation.
- Live audience captions over WebSockets.
- Custom technical vocabulary for transcription and translation.
- Independent stage pipelines.
- Session restart from the audience interface.
- JSONL event storage.
- Automatic timestamped WebVTT generation.
- Docker-based setup.
The project is available as open source under the Apache 2.0 license.
Log in or sign up for Devpost to join the conversation.