MedLive AI

• 

  An AI triage assistant that uses voice and vision to assess patient symptoms, fill out intake forms, and automatically book appointments

Inspiration

Administrative burnout is a massive issue in healthcare. Front-desk staff and triage nurses spend countless hours on the phone collecting patient symptoms, filling out intake forms, and managing calendar bookings. Meanwhile, patients often face long wait times on the phone just to get an introductory assessment and schedule an appointment. We wanted to build something that acts as a smart, autonomous front-line assistant for clinics, freeing up human workers to focus on actual patient care.

What it does

MedLive AI acts as a virtual triage nurse. Patients interact with "Dr. Liv," a lifelike AI avatar, via real-time voice and video.

1. Conversational Intake: Dr. Liv asks the patient about their symptoms, age, and contact information using natural, low-latency voice conversation.
2. Visual Assessment: Patients can show their injuries, rashes, or physical symptoms to the camera. Dr. Liv "sees" this live video feed and incorporates the visual context into her assessment.
3. Smart Triage: Based on the symptoms, the AI evaluates the urgency (Emergency, Urgent, Semi-Urgent, Routine).
4. Automated Booking: For emergency/urgent cases, Dr. Liv books an in-person calendar slot. For routine cases, she automatically generates a Google Meet link and books a virtual consultation.
5. Real-time CRM: Everything the AI hears and sees is automatically extracted and populated into a live Google Sheets CRM for the doctor to review before the appointment.

How we built it

• AI Core: Built on the Gemini 2.5 Flash model leveraging the Multimodal Live API to handle both native audio streaming and live video frame processing.
• Agent Framework: We utilized the Google GenAI SDK approach to connect to the Gemini Live API, implementing a high-performance WebRTC transport layer to manage the low-latency audio/video connection, handle conversational interruptions, and maintain the state of the consultation block.
• Avatar: Integrated Anam AI to give the agent a human-like, lip-synced visual presence.
• Frontend & Backend: A Next.js (React) application deployed on Google Cloud Run. The backend containerizes the AI agent and uses function calling to interact with the Google Calendar API and the Google Sheets API.

Challenges we ran into

Integrating the Google Calendar API within a service account environment on Cloud Run proved tricky. Service accounts lack the built-in domain-wide delegation required to send automated invite emails directly to patients or use the native hangoutsMeet conference creation in the Calendar API. We solved this by generating unique Google Meet links dynamically and modifying the agent to write the booking directly to the doctor's personal calendar via delegated access, ensuring the calendar was accurately blocked without failing.

Accomplishments that we're proud of

Proud of the seamless multimodal experience. The fact that a patient can both talk to the AI and show the AI their physical symptoms in real-time and have that instantly translated into a booked calendar appointment and a filled-out CRM row on Google Sheets feels like magic and represents a real leap forward in telehealth.

What we learned

Learned a massive amount about WebRTC, managing real-time streaming latency, and effectively using function tools within the Gemini Live framework. Handling context and ensuring the agent didn't "hallucinate" medical advice (sticking strictly to symptom collection, triage, and booking) required careful prompt engineering and system instructions.

What's next for MedLive AI

In the future, we'd like to integrate MedLive directly with standard Electronic Health Record (EHR) systems (like Epic or Cerner) via FHIR APIs, add multi-language support so non-English speakers can easily get triaged without translators, and implement an SMS notification module for appointment reminders.

Built With

• anam-ai
• css
• gcp-secret-manager
• gemini-2.5-flash
• google-calendar-api
• google-cloud-run
• google-sheets-api
• multimodal-live-api
• next.js
• python
• react
• tailwind
• typescript
• webrtc