MedDesk Copilot: AI Agents for Healthcare Communication

DEMO LINK (since loom links were not supported in the input below :p): https://www.loom.com/share/4462bbdb95d1424a92231fc547721e5d

Inspiration

I've architected pharma communication systems for two years, including an IVR for CVS handling 100M calls/year. My teammate Rafael's wife is a practicing physician who spends 3–4 hours daily on repetitive admin work: answering the same medication questions, chasing appointment confirmations, delivering post-op instructions. She told us: "I went to medical school to heal people, not repeat the same conversation 100 times." We found 80–90% of doctor-patient communications (medication instructions, scheduling, follow-ups) follow predictable patterns—perfect for AI voice automation. The insight: doctors should only handle conversations when humans are truly needed.

What It Does

MedDesk Copilot is a workspace where doctors chat with AI agents to delegate patient communication via voice calls:

Real-time timeline of patient interactions and AI call transcripts
Patient context: summary, tasks, appointments, documents, prescriptions
AI suggestion cards with Approve/Edit/Dismiss actions
Conduct all communications wth patients on behalf of the doctors and backoffice staff with approvals required at every step

Core Workflow: Doctor uses chat to assign tasks to AI agents ("Call Sarah about her medication refill") → Doctor approves action → AI agent makes voice call to patient via Vapi → Call transcript and outcome logged in timeline → Doctor reviews and provides feedback. Doctors can check patient status at a glance, prioritize high-priority cases first, and respond accordingly—AI handles the rest.

AI-Powered Data Extraction: Ask AI to "summarize John's surgery recovery" or "extract all medication changes this month"—the system aggregates data across appointments, documents, prescriptions, and call transcripts, then generates structured summaries with key dates, dosages, and clinical notes. Doctors get instant answers without manual chart review, turning hours of data hunting into seconds of insight.

Key Features: Voice-first automation (AI makes actual phone calls, transcripts in timeline) • Priority-based patient triage • Smart record aggregation & insight generation • Natural language data extraction (summarize visits, extract medication history, identify care gaps) • Analytics dashboard (time saved, deflection rate, Daytona-powered charts) • Full audit trail with PHI redaction and "View reasoning" transparency.

How We Built It

Stack: CopilotKit +AG-UI for agent UI interaction + Mastra for patient details, analytics agents and task executors + Vapi for voice agents and phone calls + Daytona for complex analytical queries using code execution + FastMCP + FastAPI + SQLite + Typescript

Architecture: React frontend (CopilotKit for chat-to-task interface + AG UI approval cards) • Python FastAPI backend • Mastra orchestrates 3 agents (Orchestrator routes intent, CommsAgent handles voice calls via Vapi, AnalyticsAgent runs Daytona jobs) • FastMCP provides unified tool contract (vapi_initiate_call, db_create_task, analytics_execute) for deterministic, auditable execution • Vapi webhook receiver syncs call transcripts into patient timeline.

Flow: Doctor chats task → Mastra orchestrator picks CommsAgent → Doctor approves → FastMCP calls vapi_initiate_call → AI calls patient → Transcript logged → Doctor reviews outcome → Feedback collected.

Challenges We Ran Into

Context passing: Built shared Mastra context store so agents receive full patient summary and history. Voice integration: Vapi webhook receiver syncs call events into timeline as voice_call_log entries. Safety vs. speed: Confidence-based logic auto-approves low-risk actions; high-stakes always need review. Non-hallucinated analytics: Daytona runs actual Python code in sandbox—real charts, not LLM guesses. Demo reliability: Careful prompt engineering balanced determinism with flexibility.

Accomplishments We're Proud Of

End-to-End Workflow: CopilotKit chat → Mastra → FastMCP → Vapi call → audit—every action traceable and clinician-approved. Doctor-in-the-Loop UX: Approve/Edit/Dismiss cards with confidence scores and reasoning traces build trust. Voice-First Automation: AI makes real phone calls from chat commands; transcripts flow into unified patient timeline—visceral "wow" moment that makes delegation feel tangible. Intelligent Record Aggregation: AI automatically synthesizes appointments, prescriptions, documents, and call logs into coherent patient summaries—ask "summarize Sarah's post-op recovery" and get instant structured insights without manual chart review. Priority-Based Triage: Doctors check patient status, focus on high-priority cases first, delegate routine follow-ups to AI. Real Analytics: Daytona proves ROI with time-saved metrics and chart generation without hallucinations.

What We Learned

MCP is a game-changer: Single tool contract keeps agents reliable and auditable. Approval UX beats accuracy promises: Doctors want to review—showing confidence + reasoning builds trust faster than claiming "95% accuracy." Voice is magical: Actual phone calls make automation feel tangible—chat-to-call workflow creates visceral delegation moment. Small intents win: Atomic actions (schedule call, check medication, confirm appointment) easier to orchestrate than monolithic prompts. Priority-first UX: Doctors think in triage—high-priority patients first, routine follow-ups delegated to AI.

What's Next

Phase 2: Reinforcement learning—track doctor edits, auto-select best prompt variants from feedback. Enhanced Intelligence: Proactive insights—AI detects care gaps before doctors ask, predictive analytics for readmission risk, automated clinical note generation from aggregated records. Production: Auth/RBAC, HIPAA security, encryption, rate limiting. More Channels: Email and SMS communication alongside voice calls. Integrations: EHR/FHIR (Epic, Cerner), calendar sync, expanded doc parsing, lab results integration for comprehensive record aggregation. Vision: Doctors spend 3 hrs/day on admin (AMA). At 85% deflection, MedDesk saves 2.5 hrs/day per doctor—across 1M US physicians, 2.5M hours/day returned to patient care. Our goal: give doctors their time back to heal people.

Built with: CopilotKit • AG-UI • Mastra • Vapi • Daytona • FastMCP