Novavo

• 

  Architecture.png

Inspiration

Every second counts in an emergency room. Overworked doctors spend precious time manually assessing and categorizing patients — time that could be spent saving lives. I wanted to build something that could act as an intelligent first responder, processing what a doctor sees and hears simultaneously to deliver instant triage guidance.

What it does

Novavo is a real-time ER triage agent powered by Gemini Live API. A doctor opens the app, speaks the patient's symptoms, and points the camera at the patient. Gemini processes both the audio and visual input simultaneously and returns:

• Urgency level — Critical, Urgent, or Standard
• Clinical reasoning — 2-3 sentence explanation
  
• Suggested immediate actions — actionable bullet points
• Voice response — Gemini speaks the triage result back in real time

Every session is automatically saved to Firestore for record keeping and audit trails.

How we built it

• Frontend — React + Vite, capturing live microphone audio (PCM 16kHz) and JPEG camera frames, streaming both via WebSocket
• Backend — Python FastAPI with a bi-directional WebSocket bridge to Gemini Live's BidiGenerateContent API
• AI — Gemini Live API (gemini-2.0-flash-live-001) with a custom system prompt enforcing structured JSON triage output
• Database — Firestore for persistent session history
• Deployment — Backend on Google Cloud Run, Frontend on Firebase Hosting

Challenges we ran into

• Handling real-time bi-directional streaming between three WebSocket connections simultaneously (browser ↔ FastAPI ↔ Gemini Live)
• Parsing structured JSON triage output reliably from Gemini's streaming response chunks
• Keeping audio and video streams synchronized without dropped frames
• Configuring CORS correctly across Cloud Run and Firebase Hosting

Accomplishments that we're proud of

• Built a true multimodal agent that processes voice and vision simultaneously in real time — not just one or the other
• Gemini's structured JSON triage output is reliable and consistent across different symptom descriptions
• Clean, production-quality codebase with proper async handling, error recovery, and session persistence
• Full end-to-end deployment on Google Cloud in a single day

What we learned

• How to use Gemini Live API's BidiGenerateContent for true real-time multimodal processing
• Building production-grade async WebSocket pipelines in FastAPI
• How Google Cloud Run handles containerized deployments with environment-based secrets
• The importance of structured output prompting for reliable AI responses in high-stakes environments like healthcare

What's next for Novavo

• Integration with hospital EHR systems for automatic record creation
• Multi-patient queue management dashboard
• Support for additional languages for global ER deployment
• HIPAA compliance layer for production use
• Expansion beyond ER — field medics, ambulances, remote clinics

Built With

• cloud
• cloudrun
• fastapi
• firebase
• firebasehosting
• firestore
• gemini
• geminiliveapi
• google
• googlegenaisdk
• javascript
• pydantic
• python
• react
• uvicorn
• vite
• websocket