Insight Health Agent

Project Image

Inspiration

Health anxiety is deeply personal for me. After experiencing panic attacks, I became increasingly aware of how little most people understand their own bodies — and how hard it is to get a quick, honest answer about a symptom without waiting days for an appointment or falling into a WebMD spiral. I wanted to build something that feels like having a knowledgeable friend with medical knowledge available 24/7: someone you can just show what's worrying you and get a real, direct response. That idea — a health companion that sees, listens, and talks back in real time — became Insight Health.

What it does

Insight Health is a live multimodal AI health assistant that combines continuous camera vision with full-duplex voice conversation. From the moment you start a session, the AI sees through your camera at 2 frames per second and listens through your microphone simultaneously — no buttons to press, no modes to switch.

You can show it a skin rash, a medication label, swollen joints, or just describe how you feel — and it responds by voice in real time with specific visual observations and honest assessments. It speaks your language (Spanish or English), adapts to your pace, and maintains full conversational context across an unlimited number of exchanges without losing track of what was said earlier.

It is not a replacement for a doctor. But it is a first line of calm, informed guidance when you don't know if what you're seeing is serious.

How we built it

The stack is intentionally lean. The backend is a Python FastAPI server that manages a persistent WebSocket connection with the browser while continuously reconnecting to the Gemini Live API for each conversational turn. Conversation history is injected into the system prompt on every reconnect, so the AI never loses context or re-introduces itself.

The frontend is vanilla JavaScript — no framework — capturing PCM16 audio via the Web Audio API and JPEG frames via a canvas element, all streamed in real time over WebSocket. The UI is designed like a Twitch stream: full-screen camera feed with chat subtitles overlaid on top, built to work on mobile over HTTPS via VS Code port forwarding.

The model powering everything is Gemini 2.5 Flash Native Audio on Vertex AI, chosen for its native real-time audio capability and multimodal understanding of both speech and live video simultaneously.

Challenges we ran into

Turn-taking and knowing when to stop talking. The hardest behavioral challenge was making the AI understand that the user controls the conversation. Out of the box, the model tended to ask multiple questions in a row without ever committing to an opinion. We had to engineer the system prompt carefully to enforce a strict OBSERVE → ASSESS → one optional question flow, making sure it leads with its assessment rather than hiding behind clarifying questions.

Recognizing conditions visually. Getting the AI to give a specific, confident visual assessment of something like a skin condition — rather than deflecting with generic disclaimers — required significant prompt iteration. The model needed explicit permission and instruction to say "this looks like X" rather than "I'm not able to diagnose."

Continuous conversation across Gemini turns. The Gemini Live API closes its session at the end of each turn. Maintaining a natural, uninterrupted conversation required building an auto-reconnect loop on the backend that preserves the last 20 exchanges and injects them as context on every new connection — invisible to the user, seamless in practice.

Camera on mobile. Browsers block camera access on non-secure origins. Solving this for local development required routing traffic through HTTPS via VS Code port forwarding, which wasn't immediately obvious.

Accomplishments that we're proud of

The conversation feels genuinely continuous. You can talk for 10+ minutes, switch topics, refer back to something said earlier, and the AI tracks it all — without ever restarting or re-introducing itself. That seamlessness was not easy to achieve and makes the experience feel truly live rather than chatbot-like.

We're also proud of the directness we achieved in the AI's responses. It gives real opinions based on what it sees. That required pushing back against the model's default tendency toward over-caution, and we think the result is meaningfully more useful for someone who just wants a straight answer.

What we learned

We learned what it actually takes to build a live agent — not just an API call with a streaming response, but a system that manages bidirectional audio, continuous video, session state, and conversational memory simultaneously in real time. The engineering surface is much larger than it appears.

We also deepened our understanding of how AI models handle visual medical content: how to describe symptoms effectively in prompts, how to guide the model toward confident visual reasoning, and where the real boundaries of visual understanding lie (lighting, angle, resolution all matter enormously).

And personally — building something in the health space forced us to think carefully about what "helpful" actually means when someone is anxious about their body. Sometimes the most valuable thing isn't a perfect diagnosis. It's a calm, direct voice that says: here's what I see, here's what it could be, and here's what I'd do next.

What's next for Insight Health Agent

Symptom timeline tracking: Remember symptoms across sessions so the AI can detect patterns over time ("your rash looks better than last week") Structured triage output: Generate a shareable summary after each session that the user can send to their actual doctor Fine-tuned visual recognition: Train on labeled datasets of skin conditions, wound types, and medication labels for more accurate identification Proactive alerts: If the camera detects something concerning the user hasn't mentioned, the AI surfaces it rather than waiting to be asked Wearable integration: Pull in heart rate, SpO2, and sleep data to correlate with what the user describes and what the camera sees