VitalVoice: AI Health Agent via MCP

Inspiration

Chronic disease patients often struggle to consistently track symptoms, remember medication schedules, and communicate health changes to their providers between visits. By the time they sit in the exam room, weeks of nuance get compressed into "I've been doing okay, I guess." Meanwhile, providers lack the structured longitudinal data they need to make informed adjustments.

We asked: what if a patient could just talk about how they're feeling, and an AI agent handled the rest? Log the symptoms, flag dangerous drug interactions, track adherence, and hand the provider a clean SOAP note — all from a conversation.

What it does

VitalVoice is a voice-first AI health monitoring agent that conducts structured patient check-ins and dynamically invokes medical tools to:

🎙️ Conversational check-ins — patients speak naturally; the agent asks follow-ups and rates severity
📊 Symptom tracking — every reported symptom is logged with severity (1–10) and timestamped
💊 Medication interaction checks — mentions of multiple drugs trigger real-time interaction lookups via OpenFDA
📈 Adherence monitoring — tracks whether patients are taking medications on schedule over a 7-day window
📝 SOAP note generation — automatically produces provider-ready Subjective/Objective/Assessment/Plan summaries
🖥️ Live clinical dashboard — a dark-themed React dashboard with trend charts, medication cards, and transcript history

Live demo → vitalvoice-eta.vercel.app

How we built it

The system has three core layers:

1. MCP Tool Servers (the brain) Each medical capability is a standalone Model Context Protocol (MCP) server running over stdio transport. We built three servers exposing 8 tools total:

Symptom Tracker — log_symptom, get_symptom_history, get_trends
Medication Checker — check_interaction, log_medication, get_adherence_report
Summary Generator — generate_soap_note, get_latest_summary

MCP makes the tools modular, discoverable, and standards-based — any MCP-compatible agent can invoke them without custom integration code.

2. FastAPI Backend (the spine) A Python FastAPI server aggregates data from the MCP servers' JSON stores, computes trend series and adherence summaries, and serves everything through REST endpoints + WebSocket for real-time updates.

3. React Dashboard (the face) A single-page clinical dashboard built with React 19, TypeScript, TailwindCSS v4, and Recharts. It polls the backend for live data and renders symptom trend charts, medication adherence cards, SOAP notes, and the full conversation transcript — all with smooth animations and a dark medical aesthetic.

4. Voice Pipeline (the voice) An optional LiveKit-based pipeline connects Deepgram Nova-3 (STT) → Groq LLM (reasoning + tool selection) → Cartesia Sonic (TTS) for real-time voice conversations. The LLM dynamically discovers and invokes MCP tools mid-conversation.

Why this fits Algofest

VitalVoice is not just a product demo — it reflects the exact values ALGOfest emphasizes.

This project required more than UI polish. It demanded:

structured reasoning over natural-language input modular tool orchestration real-time data flow scalable architecture choices usable outputs for real-world healthcare workflows

That aligns closely with ALGOfest’s focus on combining algorithmic excellence with practical impact, especially within the AI/ML and HealthTech directions highlighted by the hackathon.

Challenges we ran into

Timezone-aware datetime comparisons — mixing naive and aware datetimes caused silent failures in trend aggregation; we standardized on datetime.now(timezone.utc) everywhere and wrote a _parse_ts() helper
Vercel's 250MB Lambda limit — our full dependency tree (LiveKit, MCP SDK, Deepgram) blew past the limit; we had to split into a trimmed serverless requirements file for deploy vs. full local requirements
MCP stdio transport in serverless — MCP servers use subprocess communication, which doesn't map cleanly to serverless functions; we solved this by having the Vercel API read directly from the JSON data files while keeping MCP as the write interface
TypeScript 6 strictness — TS7 deprecated baseUrl/paths in tsconfig, breaking all path aliases; we migrated to relative imports and Vite-level aliases

Accomplishments that we're proud of

8 working MCP tools across 3 servers, all independently testable and composable
Deterministic seed data (random.seed(42)) that tells a coherent clinical story — a patient improving over 7 days with realistic symptom patterns
Drug interaction detection using both a local knowledge base and the FDA's openFDA API as fallback
Deployed and live at vitalvoice-eta.vercel.app — judges can explore the full dashboard without installing anything

What we learned

The Model Context Protocol is a powerful abstraction for building modular AI tool ecosystems — defining tools as MCP servers made them trivially reusable across different agent frameworks
Voice-first interfaces demand a different UX mindset — structured data extraction from natural language is harder than it looks
JSON file persistence is surprisingly viable for hackathon-scale demos and eliminates an entire class of deployment complexity

What's next for VitalVoice

EHR/FHIR integration — push SOAP notes directly into electronic health records
Multi-language support — patient check-ins in Spanish, Mandarin, Hindi, and more
Wearable data ingestion — pull Apple Health / Fitbit vitals into the trend dashboard
Provider portal — alert routing and multi-patient views for clinical staff
HIPAA-compliant deployment — encrypted storage, audit logging, access controls ## What we learned