DocBook

Multimodal receptionist that processes voice requests + visual inputs (cards, calendars, notes) to handle bookings seamlessly in natural conversation.

Comment

🏥 DocBook — AI-Powered Medical Appointment Booking Agent

A learning journey into building a multimodal voice agent that books doctor appointments through natural conversation — powered by Gemini Live 2.5 Flash Native Audio, Vertex AI, ADK-style function calling, and Google Cloud.

💡 Inspiration

Every year, over 1 billion phone calls are made to schedule medical appointments in the U.S. alone. The average patient spends $\sim 8.1$ minutes on hold per call, and clinics lose an estimated $\$150{,}000$ annually to no-shows and scheduling inefficiencies.

We asked ourselves:

What if booking a doctor's appointment was as easy as having a conversation?

The rise of large multimodal models — particularly Google's Gemini Live API with native audio — opened the door to something that wasn't possible before: an AI receptionist that doesn't just transcribe speech and generate text-to-speech, but natively understands and speaks like a human. No STT → LLM → TTS pipeline. No latency. Just a natural, real-time voice conversation.

As a learning opportunity, I wanted to go beyond a simple chatbot demo and build a complete, end-to-end system — from microphone to database write to email confirmation — to deeply understand how multimodal AI agents works.

🩺 What It Does

DocBook is a multimodal AI medical receptionist that handles the complete appointment booking workflow through voice:

$$ \text{Patient Voice} \xrightarrow{\text{WebSocket}} \text{Gemini Live API} \xrightarrow{\text{Function Calling}} \text{Firestore + Email} $$

Core Capabilities

  1. 🎙 Voice-First Booking — Patients speak naturally; the AI understands intent, asks follow-up questions, and books appointments — all via native audio (no STT/TTS pipeline).

  2. 📸 Insurance Card Scanning — Patients show their insurance card to the camera. Gemini Vision extracts: $$ \text{Image} \xrightarrow{\text{Gemini Vision}} {\text{patientName}, \text{policyNumber}} $$

  3. 📅 Real-Time Availability — The agent queries Firestore for a doctor's booked slots and computes available $30$-minute windows: $$ \text{Available Slots} = {t \mid t \in [09{:}00,\, 17{:}00),\ t \notin \text{Booked}},\quad \Delta t = 30\ \text{min} $$

  4. ✅ Appointment Booking — Writes confirmed appointments to Firestore with conflict checking via composite indexes.

  5. 📧 Email Confirmation — Sends a beautifully formatted HTML confirmation email via SendGrid.

  6. 💬 Live Transcription — Both patient speech and AI responses are transcribed in real-time and displayed in a chat UI alongside the voice conversation.

Supported Agent Tools (ADK-Style)

Tool Input Output
get_available_slots doctorId, date List of open $30$-min slots
extract_insurance_info image_base64 (JPEG/PNG) {patientName, policyNumber}
book_appointment doctorId, patientName, email, policy, startTime Confirmation with appointmentId
send_email_confirmation appointmentId, email {success: true/false}

🛠 How We Built It

Architecture at a Glance

$$ \boxed{\text{React PWA}} \xleftrightarrow{\text{WebSocket (PCM audio)}} \boxed{\text{Node.js Backend}} \xleftrightarrow{\text{Live API}} \boxed{\text{Gemini 2.5 Flash Native Audio}} $$

Tech Stack

Layer Technology Why
AI Model gemini-live-2.5-flash-native-audio via Vertex AI Native audio understanding — no STT/TTS latency
Live API @google/genai SDK — bidirectional streaming Real-time audio + function calling in one session
Backend Node.js 20 + Express + WebSocket (ws) Low-latency WebSocket relay + REST API
Database Cloud Firestore (Native mode) Sub-$10$ms reads, composite indexes for complex queries
Email SendGrid (via local email function) Serverless-ready email delivery
Frontend React 18 + TypeScript + Vite PWA Installable, offline-capable, responsive
Auth Application Default Credentials (ADC) Secure, token-based access to Vertex AI
Deployment Runs locally; Cloud Run–ready with Dockerfile Containerized, ready for production

Key Implementation Details

1. Audio Pipeline

The browser captures microphone audio as PCM 16-bit, 16 kHz mono and sends it as base64 over WebSocket:

$$ \text{AudioWorklet} \xrightarrow{\text{float32} \to \text{int16}} \text{Base64} \xrightarrow{\text{WebSocket}} \text{server.js} \xrightarrow{\texttt{sendRealtimeInput}} \text{Gemini} $$

Gemini responds with 24 kHz PCM audio that streams back through the same WebSocket and plays immediately via an AudioStreamPlayer.

2. ADK-Style Function Calling

When Gemini needs external data, it emits toolCall events. The backend routes these to the appropriate handler:

$$ \text{Gemini toolCall} \xrightarrow{\text{executeTool()}} \begin{cases} \texttt{get_available_slots} & \to \text{Firestore query} \ \texttt{extract_insurance_info} & \to \text{Gemini Vision} \ \texttt{book_appointment} & \to \text{Firestore write} \ \texttt{send_email_confirmation} & \to \text{Email Function POST} \end{cases} $$

The tool result is sent back via sendToolResponse({ functionResponses: [...] }).

3. Firestore Composite Indexes

Queries like "find all booked appointments for doctor X on date Y" require composite indexes:

$$ \texttt{appointments} : (\texttt{doctorId} \uparrow,\ \texttt{status} \uparrow,\ \texttt{startTime} \uparrow) $$

Without these, Firestore returns a FAILED_PRECONDITION error — a lesson learned the hard way!

🧗 Challenges We Ran Into

1. The responseModalities Gotcha

The native audio model only supports responseModalities: [AUDIO]. Attempting [AUDIO, TEXT] results in a WebSocket close with code 1007:

$$ \texttt{[AUDIO, TEXT]} \Rightarrow \text{Error 1007 (unsupported)} $$

$$ \texttt{[AUDIO]} + \texttt{outputAudioTranscription: {}} \Rightarrow \text{✅ Audio + Transcriptions} $$

Lesson: Use inputAudioTranscription and outputAudioTranscription config to get text alongside native audio.

2. The Silent Microphone — sendRealtimeInput Format

This was the most frustrating bug. The microphone was capturing audio, encoding it, sending it over WebSocket, and the backend was forwarding it to Gemini — but Gemini never heard anything. No errors. Complete silence.

The issue? The @google/genai SDK expects:

$$ \texttt{sendRealtimeInput}({ \texttt{audio}: { \texttt{data}, \texttt{mimeType} } }) $$

We were sending:

$$ \texttt{sendRealtimeInput}({ \texttt{data}, \texttt{mimeType} }) \quad \text{← silently ignored!} $$

Lesson: Always check the exact SDK payload shape. The SDK silently drops malformed input.

3. React Stale Closures

The connect callback in React was created with [] dependencies, so startMicrophone captured a stale reference. The fix: a useRef that always points to the latest function (startMicrophoneRef.current()).

4. Base64 Encoding Stack Overflow

Using String.fromCharCode(...uint8Array) crashes on large audio buffers due to the spread operator exceeding the JavaScript call stack limit. Solution: a simple for loop.

$$ \text{Spread operator} : O(n)\ \text{stack frames} \Rightarrow \text{RangeError at } n \approx 100{,}000 $$

5. Firestore Composite Index Requirements

Multi-field queries like where("doctorId", "==", X).where("status", "==", "booked").where("startTime", ">=", Y) require pre-built composite indexes. Firestore won't auto-create them — you must define them explicitly via gcloud or the console.

6. Tool Response Format

Another silent failure: sendToolResponse(array) doesn't work. The SDK expects:

$$ \texttt{sendToolResponse}({ \texttt{functionResponses}: [\ldots] }) $$

7. Model Discovery

Finding the right model identifier was a journey: gemini-2.0-flash-live-001 → gemini-2.0-flash → gemini-2.5-flash-native-audio-preview → finally gemini-live-2.5-flash-native-audio.

🏆 Accomplishments That We're Proud Of

✅ End-to-End Voice Agent — Working!

A patient can literally talk to DocBook, and the agent will:

  1. Greet them warmly
  2. Understand which doctor and date they want
  3. Query real availability from Firestore
  4. Read their insurance card via camera
  5. Book the appointment with conflict checking
  6. Send a confirmation email

All through natural voice conversation — no buttons, no forms, no menus.

✅ Sub-Second Latency

By using the Gemini Live API with native audio (no STT/TTS pipeline), the conversation feels natural:

$$ t_{\text{response}} \approx 200\text{–}500\ \text{ms} \quad \text{(vs. 2–3s with STT+LLM+TTS)} $$

✅ Multimodal Integration

Voice + Vision + Function Calling + Email — all in a single Live API session. The insurance card scanning works seamlessly within the conversation flow.

✅ Production-Ready Architecture

  • Dockerized backend ready to deploy to Cloud Run
  • Composite Firestore indexes for performant queries
  • PWA — installable on mobile and desktop
  • Service Worker for offline capability
  • Error handling with try/catch around every tool execution

✅ Built as a Solo Learning Project

Every line of code, every debugging session, every architectural decision — all done as a hands-on learning exercise to deeply understand multimodal AI agent development.

📚 What We Learned

1. Multimodal AI is a Paradigm Shift

Native audio models fundamentally change the architecture. Instead of:

$$ \text{Mic} \to \text{STT} \to \text{LLM} \to \text{TTS} \to \text{Speaker} $$

It's now:

$$ \text{Mic} \to \text{Gemini (Native Audio)} \to \text{Speaker} $$

This eliminates $\sim 2$ seconds of latency and reduces $3$ API calls to $1$.

2. The @google/genai SDK is Powerful but Subtle

The SDK's Live API is incredibly capable, but the payload formats are strict and failures are often silent. Key learnings:

  • Audio goes in { audio: { data, mimeType } }, images in { media: { data, mimeType } }
  • Tool responses wrap in { functionResponses: [...] }
  • Only [AUDIO] modality for native audio models; use transcription config for text

3. Agent Development Kit (ADK) Patterns

ADK-style function calling is essentially a tool router: Gemini decides when and which tool to call, and the backend provides structured results. The AI handles all the conversational glue.

$$ \text{Agent} = \text{LLM} + \text{Tool Declarations} + \text{System Instruction} + \text{Execution Router} $$

4. Firestore is Fast but Opinionated

Firestore's performance is excellent ($< 10$ ms reads), but its query model requires planning:

  • Composite indexes must be created ahead of time
  • Range filters (>=, <) can only apply to one field per query
  • Inequality filters on multiple fields require careful index design

5. React + Real-Time Audio is Tricky

Real-time audio in React requires careful state management:

  • AudioWorklet for microphone capture (not the deprecated ScriptProcessorNode)
  • Refs to avoid stale closures in callbacks
  • Loop-based encoding to avoid stack overflows
  • Immediate playback via AudioContext for Gemini's response audio

6. Google Cloud Ecosystem is Cohesive

Vertex AI + Firestore work together seamlessly when you use Application Default Credentials (ADC). One gcloud auth application-default login and everything connects. The project is also Cloud Run–ready with a Dockerfile for future production deployment.

🔮 What's Next for DocBook

Short-Term Enhancements

Feature Description
🌐 Multi-Language Support Hindi, Spanish, Mandarin — Gemini already understands $100$+ languages natively
📱 SMS Reminders Twilio integration for appointment reminders $24$h before
🔐 Patient Authentication Firebase Auth for returning patients with appointment history
📊 Analytics Dashboard Clinic-side dashboard showing booking patterns, peak hours, no-show rates

Medium-Term Goals

Feature Description
🏥 Multi-Clinic Support Tenant isolation for multiple clinics on one platform
🤖 Proactive Rescheduling AI detects cancellations and proactively offers slots to waitlisted patients
💊 Prescription Refill Extend the agent to handle prescription renewal requests
📋 Pre-Visit Intake Forms Agent collects symptoms, allergies, medications before the appointment

Long-Term Vision

The ultimate goal is to make every healthcare interaction as effortless as a conversation.

🧰 Built With

  • Gemini Live 2.5 Flash Native Audio — Real-time multimodal AI with native audio + vision + function calling
  • Vertex AI — Enterprise-grade model hosting with ADC authentication
  • Google Cloud Firestore — NoSQL database with composite indexes
  • Google Cloud Run — Containerized deployment target (Cloud Run–ready with Dockerfile)
  • SendGrid — Transactional email API (runs locally via functions-framework)
  • Node.js 20 + Express — Backend runtime + HTTP server
  • WebSocket (ws) — Real-time bidirectional audio streaming
  • React 18 + TypeScript — Type-safe, component-based UI
  • Vite — Fast build tooling with HMR
  • PWA (Service Worker + Manifest) — Installable, offline-capable web app

👩‍💻 About

This project was built as a learning opportunity to explore the cutting edge of multimodal AI agent development. Every challenge — from silent microphone bugs to Firestore index errors — was a lesson that deepened my understanding of how real-time AI systems works.

Built With

Share this project:

Updates

Submission history