-
-
MedVision: Real-time AI triage for faster, safer clinical decisions.
Inspiration
MedVision came from one core problem: in urgent care moments, clinicians have to process too much information too quickly. We wanted to build an AI teammate that can listen, observe, and surface critical triage signals in real time, so medical teams can focus on patients instead of juggling fragmented inputs
What it does
MedVision is a real-time triage assistant. It takes live audio and video input, streams it to an AI backend, and returns:
live transcript updates, spoken/interactive agent responses, triage cards with urgency-focused insights, a session log and downloadable report for review. In short, it acts like a multimodal “clinical co-pilot” during fast-moving assessments.
How we built it
We built MedVision as a full-stack, realtime system:
Frontend: React + TypeScript dashboard for camera feed, transcript, triage panel, and controls. Backend: FastAPI-based service that manages live sessions and AI orchestration. Realtime flow: microphone audio + periodic video frames sent to the model, then rendered as transcript and triage output. Reliability features: audio gating, interrupt support, state-aware session controls, and structured logging/report export. The focus was on low-friction UX under pressure, not just model output quality.
Challenges we ran into
Managing real-time audio without echo or feedback loops. Keeping latency low while still getting useful, stable triage output. Synchronizing multiple live states (camera, mic, speaking status, connection state, logs). Handling local/cloud environment differences during setup and deployment. Designing an interface that stays clear when many events happen at once.
Accomplishments that we're proud of
Built an end-to-end multimodal prototype that actually works live. Created a clean command-center UI that surfaces the right information quickly. Implemented practical safety/UX controls like mute, interrupt, and connection-aware settings. Added structured session logs and downloadable reports for transparency and review. Demonstrated a strong foundation for AI-assisted clinical triage workflows.
What we learned
Real-time AI UX is mostly about trust, clarity, and timing. Small engineering decisions (buffer windows, gating, retries) have huge UX impact. In healthcare contexts, explainability and traceability matter as much as speed. Building with clinicians in mind means reducing cognitive load, not adding more screens.
What's next for MedVision
Improve triage precision with richer medical prompting and better guardrails. Add multilingual voice quality improvements and locale-aware triage phrasing. Introduce clinician-in-the-loop review tools for safer recommendations. Expand integrations (EHR-ready export formats, hospital workflow hooks). Run pilot testing with real users to measure impact on triage speed and confidence.
Built With
- css
- google-cloud
- html
- lucide
- meadiadevices
- python
- react
- tailwind
- terraform
- typescript
- vite
Log in or sign up for Devpost to join the conversation.