Lingua

Built at CTP Hack Day.

Inspiration

If you've sat through a doctor's visit or a school meeting in a language you don't speak well, the hard part isn't only the words — it's the aftermath. Was the appointment the 12th or the 20th? Insurance card, photo ID, or both? Translation apps do the sentence in front of you and then forget it. We wanted the one that also takes notes.

What it does

A real-time conversation assistant for high-stakes conversations with schools, doctors, landlords, and government offices. Three screens: pick your languages, have a live translated conversation with bilingual subtitles, then get a structured summary in your language — appointments, deadlines, documents, next steps, and anything worth clarifying. Demo pair is Urdu and English.

Core rule: never invent facts that weren't said. Missing information returns an empty list, not a plausible guess.

How we built it

Frontend: React 19 + TypeScript, Vite, oxlint. Backend: Node + Express in TypeScript. Two routes — one that mints short-lived Gemini Live tokens, one that turns a transcript into schema-validated JSON. AI: Gemini Live API for streaming speech-to-speech translation, Gemini Flash with a strict schema for the summary.

The GEMINI_API_KEY never leaves the server. The browser gets an ephemeral token and connects to Gemini Live directly — and the client rejects any token missing the auth_tokens/ prefix, since the SDK would otherwise send it as a plain API key in the WebSocket URL.

The audio pipeline was most of the work: an AudioWorklet capturing 16 kHz PCM16 in 100 ms chunks off the main thread, browser-free conversion math so it's unit-testable, a playback scheduler for gap-free 24 kHz output, and a pure state machine (stopped → connecting → listening → translating) so the session can't reach an impossible state. It all sits behind one useTranslationSession() hook with idempotent start()/stop().

Process: seven scoped issues with a dependency map, one PR per issue, GitHub Actions on every PR, and an AI_USAGE.md logging what AI proposed versus what a human verified.

Challenges

The Live Translate docs say ephemeral tokens use v1beta; the SDK warns they're v1alpha only. We traced it — the warning is just a console.warn, and the SDK picks its auth method from the token prefix regardless. Also: the SDK's connect promise never settles on failure, so a socket that opens and goes quiet hangs in connecting forever until you add your own timeout.

Limitations

Urdu → English is merged; English → Urdu reuses the same architecture but isn't shipped, so it's not yet two-way. The interpreter UI isn't built — what exists is a dev harness at /?live=1. The summary is backend-only; the route and validation work, but no screen consumes it yet. End-to-end translation with real speech is unverified. We're not claiming quality we haven't measured. The deployed link isn't live; run it locally. Nothing is persisted by design — no history, no recovery from a refresh. Two languages, one laptop, headphones recommended.

What's next

English → Urdu, the interpreter and summary screens, then misunderstanding detection — flagging in real time when something didn't land, while both people are still in the room.

Team Syed Faisal Mahmood (@smahmood-data) — repo owner, setup and deployment Adil Ahmed — live audio pipeline, architecture diagrams, CI and workflow Jamis Bade (@Jawmis) — secure Gemini backend, ephemeral tokens, summary endpoint Emma Katz (@emmakatz06) — review and documentation

Slides: https://docs.google.com/presentation/d/1_Wff6cgaZJ55NGJTYi_Q2c_yNq-Xc20ApWvNN1WhSxE/edit?usp=sharing