MediSight

MediSight - Talk naturally, show your camera, get instant medical guidance. A real-time AI medical visual assistant powered by Gemini Live API

Comment 1

Inspiration

Losing a dear person is never easy. But losing someone to a condition that could have been caught early, that stays with you differently.

Someone close to me passed away from undetected kidney failure. For years, they had been living with hypertension that went unnoticed. Not because they were careless. But because access to basic primary healthcare in Nigeria is not a guarantee, it is a privilege. By the time the tests were run and the diagnosis was clear, it was too late.

They didn't have to die.

That loss never left me. And when I discovered the Gemini Live Agent Challenge, I knew exactly what I had to build.

What MediSight Does

MediSight is a real-time AI medical visual assistant that anyone can talk to naturally, instantly, and for free. No appointments. No waiting rooms. No medical jargon.

  • Medication Analysis** — Point your camera at a pill bottle and get instant dosage, side effects, and warnings
  • Prescription Reading — Show a doctor's prescription and get a plain English explanation
  • Symptom Assessment — Describe your symptoms and understand severity and next steps
  • Medical Document Translation — Show lab results and understand every number in plain English
  • Google Search Grounding — Real-time web search ensures answers are based on current verified medical sources

How I Built It

MediSight is built on Google's Gemini Live API using gemini-2.5-flash-native-audio-latest - the most powerful real-time multimodal AI available.

Architecture:

  • Frontend: Vanilla JS, Web Audio API, AudioWorklet for VAD and mic capture
  • Backend: Python 3.11, FastAPI, WebSocket streaming
  • AI: Gemini 2.5 Flash Native Audio via Google GenAI SDK
  • Grounding: Google Search grounding for real-time verified medical information
  • Deployment: Google Cloud Run

The browser captures microphone audio at 16kHz PCM and camera frames as JPEG images, streamed over WSS WebSocket to a FastAPI backend on Google Cloud Run, which relays everything to the Gemini Live API bidirectionally.

Challenges I Faced

1. Real-Time Interruption Handling The hardest feature to get right. When the user speaks while MediSight is responding, she must stop instantly. The solution uses Gemini's native VAD with an interruptPending gate that blocks buffered audio chunks from replaying after stopAllAudio() is called.

2. The ScriptProcessorNode Bug Early VAD was unreliable because Chrome throttles the deprecated ScriptProcessorNode. Migrating to AudioWorkletNode — which runs on a dedicated OS audio thread - gave us 125 VAD readings per second vs 10, making interruption rock solid.

3. VAD Threshold Tuning A threshold that works in a quiet room breaks in a normal home. After extensive real-world testing, 0.25 RMS was the sweet spot that ignores background noise but catches the voice instantly.

4. WebSocket Security on HTTPS Deploying to Cloud Run exposed a ws:// vs wss:// mismatch. Fixed by detecting the page protocol dynamically and switching to WSS automatically.

What I Learned

  • Real-time audio engineering is genuinely hard - AudioWorklet vs ScriptProcessorNode is the difference between a broken product and a professional one
  • Grounding is essential for medical AI - Google Search grounding ensures MediSight answers from current verified sources
  • Persistence beats perfection - every error had a solution
  • The Gemini Live API is extraordinarily powerful for building real human experiences

Accomplishments

  • Professional interruption handling - AI stops instantly when the user speaks
  • Gapless 24kHz audio playback with Web Audio scheduler
  • Auto-reconnect session management with exponential backoff
  • Google Search grounding for real-time verified medical information
  • Live camera vision - Gemini sees and analyses what you show it
  • Deployed globally on Google Cloud Run

What's Next

  • Voice language selection - Hausa, Igbo, Yoruba, French, Pidgin
  • Medical document upload - PDF lab results analysed instantly
  • Mobile app version
  • Partnership with healthcare providers in Nigeria and Africa

I created this content for the purposes of entering the Gemini Live Agent Challenge hackathon.

What it does

How I built it

Challenges I ran into

Accomplishments that I'm proud of

What I learned

What's next for MediSight

Built With

Share this project:

Updates

Submission history