Live Medical Interpreter

Real-time bilingual medical interpreter using Gemini Live API. Doctors speak, patients hear translation. Supports Multi-languages, AI summaries, and PDF certificates.

Comment

Inspiration

Working in healthcare tech in Southeast Asia, I witnessed a critical gap: doctors and patients who cannot communicate due to language barriers. Professional medical interpreters are expensive and not always available. Migrant workers, refugees, and tourists often receive suboptimal care simply because they cannot describe their symptoms or understand medical advice.

When Google released the Gemini Live API with native audio support, I saw an opportunity to build something that could genuinely help: real-time, bidirectional voice translation without the latency of traditional speech-to-text-to-translation-to-speech pipelines.

What it does

Live Medical Interpreter enables real-time voice translation between doctors and patients. The doctor speaks in English, the patient immediately hears the translation in their language (Thai, Burmese, Khmer, Lao, Vietnamese, or Chinese), and vice versa.

After the consultation, the system generates a structured summary with clinical grounding verification (a second AI model cross-checks the summary against the original transcript to prevent hallucinations). Doctors can review, edit, and generate bilingual PDF medical certificates.

How we built it

The core is Gemini 2.5 Flash with Native Audio via the Live API. Audio streams bidirectionally over WebSocket. The client captures microphone input as PCM 16kHz, sends it to the server, which forwards it to Gemini Live. Gemini returns translated audio at 24kHz which broadcasts to the other participant.

Stack:

  • Frontend: React 19 + TypeScript + Vite + Web Audio API
  • Backend: Node.js + Express + WebSocket
  • AI: Gemini 2.5 Flash Native Audio (Live API) for real-time translation
  • Speech: Google Cloud Speech-to-Text V2 (Chirp 3) for transcription
  • Storage: Firebase Firestore + Cloud Storage
  • Deployment: Google Cloud Run

Challenges we ran into

Speaker attribution in shared rooms: The biggest challenge was the agent incorrectly attributing patient speech to the doctor (and vice versa) since both are in the same physical room. Audio from the patient's device would sometimes pick up the doctor speaking, causing the agent to translate in the wrong direction. This required extensive system prompt engineering and prompt tuning through trial and error. I had to carefully instruct the agent about speaker context, add explicit role boundaries, and tune the prompts over many iterations. This took most of the development time. It still occasionally makes mistakes, but works correctly most of the time now.

Audio encoding consistency: Gemini Live outputs PCM at 24kHz, but browser AudioContext defaults vary. I had to carefully manage sample rate conversion and ensure consistent LINEAR16 encoding throughout the pipeline.

Language detection accuracy: Initial transcriptions sometimes detected the wrong language. I solved this by storing translated audio separately, then using Cloud Speech-to-Text V2 with Chirp 3 for final transcription, which has better multilingual support.

Hallucination prevention: Medical summaries cannot contain invented information. I implemented a two-model verification system where a second Gemini model reviews summaries against original transcripts and flags any unsupported claims.

Accomplishments that we're proud of

  • Achieving sub-second translation latency for natural conversation flow
  • Building a complete end-to-end workflow from live interpretation to PDF certificate generation
  • Implementing clinical grounding verification to prevent AI hallucinations in medical context
  • Supporting Multi-languages including low-resource languages like Burmese, Khmer, and Lao
  • Creating a room-based system where doctors create sessions and patients join with simple 6-digit codes

What we learned

  • The Gemini Live API is remarkably capable for real-time audio applications with low enough latency for natural conversation
  • Medical AI applications require extra guardrails; clinical grounding verification adds latency but is essential for safety
  • Audio streaming over WebSocket requires careful attention to buffer management and sample rate handling
  • Prompt engineering for medical contexts needs explicit instructions to avoid speculation and flag uncertainty

What's next for Live Medical Interpreter

  • Add more languages (Japanese, Korean, Indonesian)
  • Implement speaker diarization for better transcript attribution
  • Build a mobile app for easier clinic deployment
  • Explore on-device processing for areas with poor connectivity
  • Partner with clinics for real-world pilot testing

Built With

Share this project:

Updates

Submission history