Tongue — live multilingual voice rooms

Live voice rooms where everyone speaks their own language and hears everyone else in theirs — real-time per-listener AI dubbing at ~1.3s on AWS (Transcribe, Translate, Polly, DynamoDB).

Comment

Inspiration

The "Babel fish" has been a sci-fi promise for forty years, yet group voice chat still forces everyone onto one shared language. Captions let you read along — but you still can't just talk. We wanted a room where five people speak five languages at once and everyone simply understands.

What it does

Tongue is a live voice room where everyone speaks their own language and hears everyone else dubbed into theirs, in real time (~1.3 s). For every speaker, each listener gets the audio re-voiced into their chosen language, plus a rolling transcript (original on top, translation below). Switch your listening language and the whole room re-dubs for you on the fly; repeated phrases come back instantly from cache.

How we built it

  • Frontend: Next.js on Vercel — room UI, mic capture, per-listener language, live transcript.
  • Transport: LiveKit Cloud (WebRTC) for multi-party audio + a data channel for transcripts.
  • Dubbing agent: a Node service on AWS ECS Express (Fargate) that joins each room and subscribes to every speaker's track.
  • AI pipeline (all AWS): Amazon Transcribe (streaming STT) → Amazon Translate → Amazon Polly (neural TTS) — one pipeline per active listener language, in parallel — published back as a dub:<lang> track.
  • Data: one Amazon DynamoDB single table holds rooms, participants, the live transcript, and a translation cache keyed by sha256(phrase).

Why DynamoDB

A single-table design serves every access pattern with Query/GetItem — no scans, single-digit-millisecond reads, on-demand capacity, nothing to manage. The cache fans out by language, not by user, so a thousand listeners cost about the same as one, and repeated phrases are instant. That's what makes Tongue built for millions of rooms from day one.

Challenges we ran into

  • Latency. A throwaway spike measured the cold Transcribe→Translate→Polly pipeline at ~1.35 s end-to-end — under our 3 s bar — which greenlit the whole concept.
  • TLS in the container. The native LiveKit engine validates TLS against the OS trust store, and node:22-slim ships without CA certificates — every outbound call to AWS/LiveKit failed in production until we added ca-certificates to the image.
  • A join race. Two browser windows joining the same room fired two near-simultaneous agent summons → two agents → doubled dubs. Fixed by reserving the room slot synchronously (check-then-act race).

Accomplishments

It's real and deployed end to end — a live URL backed by a Fargate agent and DynamoDB, not a mockup.

What's next

Voice-cloning to preserve each speaker's timbre, translating on partial transcripts to shave more latency, and multi-region agents for global rooms.

Built With

Share this project:

Updates