Project ELAH

Patients hide the truth from doctors. They tell ELAH. A judgment-free companion that listens and delivers structured clinical handoffs. Enhancing AMIE and MedGemma.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Inspiration

The Diagnostic Crisis

Every year, 795,000 Americans are killed or permanently disabled by diagnostic errors — 371,000 deaths and 424,000 permanent disabilities annually. This $38.8 billion catastrophe accounts for 35.2% of all medical malpractice payouts in the United States, making misdiagnosis the single deadliest category of medical error.

The top 15 misdiagnosed conditions alone cause over 400,000 cases of serious harm per year. Stroke, sepsis, pneumonia, venous thromboembolism, and lung cancer account for 38.7% of all diagnostic deaths. The error rate for spinal abscess reaches 62.1%. If these rates were cut in half, up to 200,000 lives could be saved annually.

Sources: Newman-Toker et al., "Burden of Serious Harms from Diagnostic Error in the USA," BMJ Quality & Safety (2023); Newman-Toker et al., "25-Year Summary of US Malpractice Claims for Diagnostic Errors 1986–2010," BMJ Quality & Safety (2013), Johns Hopkins Armstrong Institute.

The Root Cause: Human Silence

The root cause isn't a lack of medical knowledge. It's a systemic failure of human communication.

Between 60% and 80% of patients have withheld medically relevant information from their doctors. Among younger adults (under 40), the numbers are staggering:

  • 81.8% omit information because they don't want to be judged or lectured

  • 75.7% don't want to hear how harmful their behavior is

  • 60.9% feel deep embarrassment about their bodies or choices

  • 50.8% fear being labeled a "difficult patient" in their medical record

  • 45.2% stay silent to avoid taking up the doctor's limited time

And this isn't just about honesty in the office — 36.4% of American adults avoid going to the doctor entirely, even when they suspect they need care. For feared conditions like Alzheimer's (41%) or Huntington's (40%), avoidance rates are even higher.

A global meta-analysis across 25 countries and 92 studies (564,000+ participants) found that nearly 1 in 3 people actively avoid receiving essential medical information.

Sources: Levy et al., "Prevalence of and Factors Associated With Patient Nondisclosure of Medically Relevant Information to Clinicians," JAMA Network Open (2018); Taber et al., "Prevalence, Predictors and Determinants of Medical Information Avoidance: A Systematic Review and Meta-Analysis," PMC (2025).

Why AI Changes Everything

Meanwhile, over 40 million people seek medical answers from AI every single day. Over 70% of these health queries happen outside office hours — when no doctor is available.

  • 13.1% of US youth (ages 12–21) already use AI for mental health counseling; among 18–21 year-olds, it's 22.2% (JAMA, 2025)

  • 61% of people globally would accept AI for psychological counseling

  • 28% of young American adults prefer bringing an emotional problem to AI rather than a human

  • Men are 42% more likely to seek emotional support from a machine than from another person (UK study)

The AI companion market has grown 700% between 2022 and 2025, projected to reach $552 billion by 2035. Character.AI alone has 20 million monthly users averaging 93 minutes per day — more time than most people spend with their therapist in a month.

The world has already decided to trust the machine with its secrets. ELAH captures that trust and hands it — structured, verified, and actionable — to the physician who can save the life.

Sources: OpenAI Health Report (2025); JAMA Study on Youth AI Mental Health Usage (2025); Kantar Global AI Survey (10,000 respondents); Character.AI usage data (2024); Grand View Research / Allied Market Research projections.

What it does

ELAH (Empathetic Live AI for Healthcare) is a two-phase psychiatric consultation platform that uses Gemini Live API to create a "minimal clinical safe space" — a voice-first AI companion that listens to what patients are too afraid to tell their doctors, then transforms that raw conversation into structured clinical intelligence for the physician.

Phase A — Patient Voice Session

The patient has a natural, real-time voice conversation with ELAH using Gemini Live API with native bidirectional audio. No forms. No text boxes. Just talking.

  • Natural conversation — Native audio streaming, not speech-to-text-to-LLM. Sub-second latency.

  • Barge-in support — Patient can interrupt naturally via Voice Activity Detection.

  • Affective dialog — ELAH adapts tone to the patient's emotional state in real time.

  • Multimodal input — Patient can show visible symptoms via camera during the session (image upload through WebRTC byte stream → Gemini Vision analysis).

  • Therapeutic persona — ELAH never judges, never diagnoses. It only listens and acknowledges. This is what breaks the stigma barrier.

  • Server-side transcript — Every conversation turn is captured via LiveKit RPC for Phase B.

Phase B — Doctor Clinical Cockpit

When the session ends, the transcript is automatically fed into a multi-phase clinical reasoning pipeline powered by Gemini 2.5 Flash:

Phase 1 — Structured Extraction (parallel processing)

Three Gemini calls run simultaneously:

  1. Clinical Extraction — Symptoms, medications, history, risk factors → Zod-validated schema (zero hallucinated medical data)

  2. SOAP Document — Structured clinical notes in the universal Subjective/Objective/Assessment/Plan format

  3. Action Center — AI-recommended next steps organized by category (prescriptions, referrals, follow-ups, communications)

Phase 2 — ELAH ToolLoopAgent (AI SDK v6)

A multi-tool AI agent that autonomously executes read-only operations and pauses for physician approval on clinical actions:

  • Auto-execute tools: check_calendar (find available slots), fetch_patient_history (retrieve diagnoses, medications, past consultations), search_drug_interactions (verify medication safety)

  • Gate tools (require explicit physician approval): send_whatsapp (message to patient's family), book_appointment (schedule follow-up), sign_prescription (generate digital prescription), update_medical_record (modify electronic health record)

The physician sees everything streamed in real-time with chain-of-thought reasoning visibility. AI proposes, doctor disposes. Zero autonomous medical decisions.

How we built it

Voice Agent (Gemini Live API)

The voice agent runs on @livekit/agents with @livekit/agents-plugin-google, connecting directly to Gemini 2.5 Flash Native Audio (gemini-2.5-flash-native-audio-preview). This is true bidirectional audio streaming — not the typical speech-to-text → LLM → text-to-speech pipeline. The model hears and speaks natively.

Key implementation details:

  • Affective dialog enabled (enableAffectiveDialog: true) so ELAH adapts its voice tone to the patient's emotional state

  • Image upload via byte stream — When the patient mentions something visual, the agent proactively calls offer_image_upload which triggers an RPC to the frontend, activating the camera upload UI. The image is sent as a WebRTC byte stream, converted to base64, and processed by Gemini Vision in-context.

  • Transcript accumulation — Every conversation turn is captured server-side via ConversationItemAdded events. The frontend retrieves the complete transcript via getSessionTranscript RPC when the session ends.

Clinical Reasoning Pipeline (AI SDK v6)

The /api/reasoning endpoint uses Vercel AI SDK v6 with two distinct flows:

  • First request (transcript arrives): Runs Phase 1 extraction pipeline (3 parallel generateText calls with Output.object for structured output), then merges the ELAH ToolLoopAgent stream into the same response using createUIMessageStream + writer.merge.

  • Subsequent requests (doctor approving/rejecting tools): Routes directly to createAgentUIStreamResponse with the ELAH agent, which continues its tool loop.

The gate tool pattern (tools defined without execute) is what creates the physician-approval workflow: the AI SDK automatically pauses the agent loop when it encounters a tool without an executor, sending an input-available event to the frontend. The doctor reviews, approves or rejects, and the loop continues.

Frontend (Next.js 16 + React 19)

The unified /elah-session page manages both phases with state transitions: idle → connecting → elah_active (Phase A) → capturing → doctor_review (Phase B).

  • SanctuaryOrb — A custom WebGL/GLSL animated orb (DesignAuraOrb) that responds to the agent's audio volume in real-time via LiveKit track hooks."

  • 15+ artifact card components — Prescription builder (with poka-yoke safety patterns), approval cards, clinical summary, medication interactions, psychic exam, evaluation scales (PHQ-9, GAD-7), action cards for referrals, WhatsApp, appointments

  • Type-safe agent UI — InferAgentUIMessage<typeof elahAgent> ensures compile-time safety between backend tool definitions and frontend rendering

Infrastructure

  • Cloud Run — Two services deployed: sanctuary-app (Next.js) and sanctuary-agent (LiveKit voice agent)

  • Artifact Registry — Docker images stored and versioned by git SHA

  • Automated deployment — Single deploy.sh script handles everything: Docker auth, Artifact Registry repo creation, image build/push, Cloud Run deploy with environment variables

Challenges we ran into

  1. Gemini Live + LiveKit integration — The @livekit/agents-plugin-google beta API for RealtimeModel had minimal documentation. We had to verify every import, parameter name, and event signature against the actual source code rather than relying on examples.

  2. Two-stream merge — Phase 1 (manual createUIMessageStream) and Phase 2 (agent stream from createAgentUIStream) needed to merge into a single HTTP response. The writer.merge() pattern was the solution, but required careful ordering to avoid race conditions.

  3. Gate tool UX — Designing the approval flow so the doctor understands exactly what the AI wants to do, can review all parameters, and approve/reject with confidence. The prescription builder needed poka-yoke safety patterns (ReviewCheckbox resets when dose/unit/duration changes) to prevent accidental approvals.

  4. Affective dialog calibration — Balancing ELAH's empathetic persona so it feels warm and safe without crossing into clinical territory. The system prompt had to be extremely precise: "You are NOT a doctor. Never diagnose. Only listen and acknowledge."

Accomplishments that we're proud of

  • Triple Gemini usage in a single product: Text (clinical reasoning), Native Audio (bidirectional voice), and Vision (image analysis) — demonstrating the full breadth of Gemini's multimodal capabilities

  • Zero autonomous medical decisions — Every clinical action requires explicit physician approval. AI proposes, doctor disposes. This isn't just a feature, it's an ethical architecture.

  • Sub-second voice latency — Native audio streaming means the patient experience feels like talking to a real person, not waiting for a chatbot to process text

  • Production-grade clinical output — Zod-validated schemas ensure that extracted medical data is deterministic and structured, not free-text hallucinations

  • Type-safe agent pipeline — ToolLoopAgent + InferAgentUIMessage provides compile-time guarantees from backend tool definitions to frontend rendering

What we learned

  1. Gemini Live API is transformative for healthcare — The difference between speech-to-text-to-LLM and native audio is not incremental. It's categorical. Patients respond to voice with a naturalness that text interfaces can never achieve.

  2. The "no-execute" pattern is powerful — Defining tools without an execute function creates a natural approval gate. The AI SDK pauses the loop automatically, and the frontend renders approval cards. This pattern should be standard for any high-stakes AI application.

  3. Affective dialog changes everything — When ELAH adapts its tone to match the patient's emotional state, disclosure rates increase dramatically. The patient forgets they're talking to a machine.

  4. Safety requires architecture, not just prompts — Poka-yoke patterns (auto-resetting checkboxes), gate tools, Zod-validated schemas — safety isn't a prompt instruction, it's a structural guarantee.

What's next for ELAH

  • Real EHR integration — Connect gate tools to actual electronic health record systems (currently mocked for the hackathon)

  • Multi-language support — Expand beyond Portuguese to serve diverse patient populations

  • Longitudinal patient profiles — Track patient disclosure patterns across sessions to build trust over time

  • Clinical validation study — Partner with psychiatric clinics to measure the actual impact on diagnostic accuracy and patient disclosure rates

  • HIPAA/LGPD compliance layer — End-to-end encryption, audit logging, and consent management for production deployment

Built with

  • Gemini 2.5 Flash (Text + Structured Output)

  • Gemini Live API (Native Bidirectional Audio)

  • Gemini Vision (Image Analysis)

  • Google Cloud Run (2 services)

  • Google Artifact Registry

  • Google GenAI SDK (@google/genai)

  • AI SDK v6 (ai, @ai-sdk/google)

  • LiveKit Agents SDK (@livekit/agents, @livekit/agents-plugin-google)

  • Next.js 16 + React 19

  • TypeScript

  • Zod (schema validation)

  • Tailwind CSS v4

  • shadcn/ui

Google Cloud Services Used

| Service | Purpose | Evidence |

|---------|---------|----------|

| Gemini API (Text) | Clinical reasoning, structured extraction, SOAP generation | src/app/api/reasoning/route.ts — google("gemini-2.5-flash") |

| Gemini Live API (Native Audio) | Bidirectional voice agent with affective dialog | src/agent/agent.ts — google.beta.realtime.RealtimeModel |

| Gemini Vision | Patient image analysis during voice session | src/agent/agent.ts — byte stream handler with llm.createImageContent |

| Cloud Run | Hosting Next.js app + voice agent (2 services) | deploy.sh — gcloud run deploy sanctuary-app / sanctuary-agent |

| Artifact Registry | Docker image storage | deploy.sh — us-central1-docker.pkg.dev |

Live deployment: https://sanctuary-app-618728668519.us-central1.run.app

Try it yourself


git clone https://github.com/Michsantozz/sanct.git

cd sanct

npm install

cp .env.example .env.local

# Add your GOOGLE_API_KEY, LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET

npm run dev          # Terminal 1: Next.js

npm run agent:dev    # Terminal 2: Voice agent

Open http://localhost:3000/elah-session

Accomplishments that we're proud of

What we learned

What's next for Project ELAH

Built With

Share this project:

Updates

Submission history