caption-anything

Every voice, instantly captioned. Real-time subtitles for real-life conversations, in any language.

Comment

Inspiration

It started in a noisy room. A group of friends talking, laughing, interrupting — and one person quietly nodding along, missing half of it. Then i thought about reels playing in cafés, meetings where people say "wait, what did you say?", and everyone with hearing loss who fakes their way through group conversations. TV has subtitles. YouTube has subtitles. Real life doesn't. I wanted to build that missing layer: subtitles for the real world.

What it does

Caption Anything turns any live conversation into real-time, color-coded captions on your screen — like subtitles for real life. Point your phone's mic at any room and every voice becomes readable text instantly:

  • **Live captions— words appear as they're spoken (interim captions render immediately, final results lock in cleanly)
  • **Speaker color-coding — diarization gives each person their own color, so you can follow who said what
  • **Any language — 50+ languages supported; select the language or use auto-detect
  • **Instant translation — non-English speech is translated to English in real time
  • **Tap to replay— tap any caption to hear it spoken again
  • **Automatic recap — one click summarizes the conversation with action items It helps people with hearing loss, non-native speakers, and anyone stuck in a noisy room — no one has to miss a word again.

How I built it

I chose a streaming-first architecture so captions appear live, not after the conversation ends.

  • **Frontend: React + Vite. The mic is captured in the browser, converted to 16-bit PCM at the device's native sample rate, and streamed over WebSocket. A live waveform visualizer and color-coded caption bubbles make the product feel alive.
  • **Backend: Node + Express. A thin WebSocket server proxies the audio stream to Deepgram's Nova-3 speech-to-text model with speaker diarization, so each voice gets its own color.
  • **Multilingual + translation: Deepgram handles 50+ languages with automatic detection for 10 major languages. Final captions are passed through Groq (free-tier LLM) to translate any language into English in real time.
  • **Recap:A "Summarize" button sends the full transcript to Groq, which returns a punchy recap with action items.

Challenges I ran into

  1. *Latency is everything.If captions lag more than a second, it feels broken. We learned that interim (partial) results must render instantly while final results lock in — and that *not blocking every phrase on a translation round-trip makes English captions feel instant.
  2. **Multilingual is not "one setting. We assumed auto-detect would handle Telugu — it didn't. Deepgram's language=multi only covers 10 languages, so we added an explicit language selector that unlocks the full 50+ language list.
  3. *Environment debugging: Our server silently failed to load .env keys depending on which directory it was launched from, and a stale server process kept serving old code on port 3001. Two hours of "why does nothing work?" taught us to check the running process and the logs *first.
  4. **Diarization in fast speech — assigning stable speaker labels through interruptions took a majority-vote heuristic over word-level speaker tags.

Accomplishments that i proud of

  • It works live.Not a demo video, not a mock — real audio streams from the mic, through our server, to the speech model and back as captions in under a second. Watching our own conversation appear as color-coded text on screen was a genuine "wow" moment.
  • True multilingual support. We didn't settle for English-only. Caption Anything transcribes in 50+ languages and translates to English in real time — so a Telugu speaker and an English speaker can follow the same conversation together.
  • Speaker separation. We give every voice its own color using Deepgram's diarization, plus a majority-vote heuristic to keep speaker labels stable through interruptions — solving a problem that stumps a lot of pro products.
  • Speaker recap with action items. Beyond captions, one button distills the whole conversation into a summary with action items.
  • Polished, demo-ready UI. A cinematic landing page, live waveform, pulsing "listening" indicator, and tap-to-replay captions — we built it to be shown off.
  • Real-time engineering discipline. We learned and applied the hard parts of streaming: PCM conversion, WebSocket proxying, interim vs. final result handling, and latency management — in under 48 hours.
  • Squeezed cost to zero. We ran the entire pipeline on free tiers — Deepgram's free credit for speech-to-text and Groq's free tier for translation and summarization. Want a shorter version?

What I learned

  • Real-time systems are about gracefully degrading: interim text now, polished text a moment later.
  • The "wow" feature (live captions) is 70% of the project; the last 30% is making failure visible and recoverable.
  • Accessibility isn't a niche — it's the best kind of product requirement, because it makes the app better for everyone.

What's next for caption-anything

Speaker names, offline on-device transcription for privacy, and captions that auto-translate while you listen instead of after each sentence.

Built With

Share this project:

Updates

Submission history