MedFirstAi

• 

• 

MedFirstAi — Emergency Scene Assessment Agent

Inspiration

Every year, thousands of lives are lost not because help wasn't coming, but because bystanders didn't know what to do in those critical first minutes. The ambulance response time can be higher, but brain damage from cardiac arrest begins in just 4-6 minutes.

We asked ourselves: What if AI could bridge that gap?

The inspiration struck during a first aid training session. The instructor said something profound: "The biggest barrier to saving lives isn't skill, it's confidence." People freeze because they're afraid of doing the wrong thing. They need a calm voice guiding them through the chaos.

That's when we envisioned MedFirstAi — an AI assistant that sees what you see, hears what you say, and guides you step-by-step through emergency procedures while you wait for professional help.

What it does

MedFirstAi is a real-time emergency scene assessment agent powered by Google's Gemini 2.5 Flash with native multimodal capabilities. Here's how it works:

Point your camera at the emergency scene Talk naturally; describe what happened, ask questions Receive real-time guidance; the AI analyzes the visual scene and provides step-by-step first aid instructions The AI can:

Assess visible injuries (bleeding, burns, positioning) Guide CPR with proper chest compression depth and rhythm Walk you through consciousness checks (AVPU scale) Provide wound care instructions based on what it sees Adapt its guidance as the situation evolves Critical Design Principle: MedFirstAi always reminds users to call 999 first. It's designed to assist while you wait for help, never to replace emergency services.

How we built it

Architecture: User Input -- WebSocket --> FastAPI Backend -- Gemini Live API --> AI Response -- Audio Stream --> User

Frontend (Next.js 14 + TypeScript):

Real-time video capture using getUserMedia() API WebSocket connection for bidirectional audio/video streaming PCM audio encoding at 16kHz for Gemini compatibility Responsive UI with Tailwind CSS

Backend (FastAPI + Python):

WebSocket relay between frontend and Gemini Live API Base64 video frame encoding (JPEG, 1 FPS) Real-time audio transcoding (PCM ↔ playback) Stateless design for horizontal scaling AI Core (Gemini 2.5 Flash) Native multimodal input (video + audio simultaneously) System prompt engineering for medical guidance persona Real-time streaming responses for low-latency interaction

The Magic: Native Multimodality Unlike traditional approaches that chain separate vision and language models, Gemini's native multimodal architecture processes video and audio in a single forward pass:

Response=f(Video Frames,Audio Input,Context)

This unified approach means the AI doesn't just "see" and "hear" separately , it understands the scene holistically, just like a human would.

Challenges we ran into

1. Audio Synchronization Hell The first prototype had a 2-3 second audio delay that made real-time conversation impossible. We discovered the issue: double Base64 encoding in the audio pipeline. The backend was encoding already-encoded data, causing buffer bloat.

Solution: Careful tracing of the data pipeline and ensuring single-pass encoding.

1. WebSocket Connection Stability Emergency situations are stressful enough without connection drops. We implemented: Automatic reconnection with exponential backoff Connection state management in React hooks Graceful degradation when network quality drops

2. Prompt Engineering for Safety Medical AI requires extreme care. We iterated extensively on the system prompt to ensure: Clear disclaimers about calling emergency services first No diagnosis, only observational guidance Conservative advice that won't cause harm Calm, reassuring tone under pressure

3. Video Frame Rate vs. Quality Tradeoff Higher FPS = more context for the AI, but also more bandwidth and latency. We settled on 1 FPS with JPEG compression, enough visual information for scene assessment without overwhelming the connection.

Optimal FPS= arg(n) min (α⋅Latency(n)+β⋅Bandwidth(n)−γ⋅Quality(n))

Accomplishments that we're proud of

🎯 Sub-Second Response Latency We achieved <1 second end-to-end latency from camera frame capture to AI voice response. In emergencies, every second counts, our architecture delivers guidance fast enough to feel like a real-time conversation with an expert.

🔊 True Hands-Free Operation The entire experience is voice-controlled. No tapping screens, no scrolling through instructions, just point your camera and talk. When you're applying pressure to a wound or performing chest compressions, your hands can't be on a phone.

🧠 Zero Hardcoding, Pure AI Intelligence We didn't build a decision tree or rule-based system. MedFirstAi uses Gemini's native multimodal understanding to analyze each scene dynamically. Show it a burn, a cut, or an unconscious person, it adapts its guidance based on what it actually sees, not what we pre-programmed.

🛡️ Safety-First Prompt Engineering After dozens of iterations, we crafted a system prompt that:

Never claims to diagnose; only observes and guides Always prioritizes 999; emergency services come first Stays calm under pressure; reassuring tone even in chaos Knows its limits; redirects to professionals when needed

🌐 Production-Ready Deployment This isn't just a demo, MedFirstAi is deployed and accessible

What we learned

Multimodal AI is transformative: The ability to process video and audio natively opens possibilities that weren't feasible with chained models. Real-time systems are hard: Every millisecond of latency matters when someone's life is at stake. We gained deep appreciation for streaming architectures. Prompt engineering is an art: Crafting a persona that's helpful, calm, and safety-conscious took more iterations than the actual code. Accessibility is non-negotiable: Hands-free voice interaction isn't a nice-to-have — in emergencies, your hands are busy saving lives.

What's next for MedFirstAi

1- Offline mode with on-device models for areas with poor connectivity 2- Multi-language support for diverse communities 3- Integration with emergency services to relay scene information to dispatchers 4- Training mode for practicing first aid without a real emergency

Built With

• fastapi
• gemni
• gemni2.5flash
• google
• netlify
• next.js
• render
• tailwindcss