ChartTalk

• 

• 

• 

• 

• 

• 

Inspiration

Every family has been there — your parent is in the hospital and you're calling the nurse's station for updates, stuck on hold, getting numbers you don't understand. Or you show up to a new specialist and spend 15 minutes filling out intake forms from memory, misspelling drug names, forgetting a diagnosis from three years ago. The data already exists in the hospital's system. Since 2022, every US hospital is legally required to make it available through standardized FHIR APIs. The pipes are there. Nobody built the family layer on top. That's ChartTalk.

What it does

ChartTalk is a family health app that connects your family's hospital records in one place and lets you interact with them by voice. Connect Mom's chart, Dad's chart, grandma's chart — then ask about anyone: "what medications is grandma on?" or "are Dad's labs normal?" The AI pulls real data from their records and answers in plain English, speaking the response back to you.

Beyond querying records, ChartTalk has a symptom charting mode. Say "I want to log my symptoms" and the AI walks you through a structured intake — chief complaint, onset, duration, severity, associated symptoms, aggravating and relieving factors, prior treatment — one question at a time. It saves everything as a shareable chart summary with body region mapping, so you can send it to your doctor before your visit. Wrong department? They can refer you immediately without a wasted appointment.

How we built it

We built ChartTalk on Next.js 16 deployed to Vercel, using App Router with server-side rendering for share pages. The EHR connection uses SMART on FHIR OAuth with PKCE against Epic's sandbox, providing a real standards-based connection that automatically parses and normalizes FHIR R4 resources into Supabase.

The conversational AI runs on Grok 3 Mini (xAI) with function calling. We defined 11 tools that let the model query patient records (conditions, medications, labs, vitals, allergies, encounters) and run structured symptom charting sessions. For voice, we use xAI STT for speech-to-text and xAI TTS for text-to-speech, with sentence-level streaming for low-latency responses.

Supabase handles all persistence: patients, conditions, medications, observations, encounters, allergies, chart sessions, chart entries, and share tokens with 7-day expiry. We also built an interactive body map using SVG regions with SNOMED CT code mapping for symptom location tracking, and a share system with UUID-based tokens supporting two view types: full records and chart summaries.

Challenges we ran into

Epic's app registration has a ~1 hour sync delay after changes, which burned time during initial setup. The app was accidentally released for production, locking the incoming API selections — we couldn't go back to add Procedure.Search.

FHIR resource normalization was trickier than expected. Medication data can come as medicationCodeableConcept, medicationReference, or plain text, and we had to handle all three. Allergy entries include "no known allergies" sentinel codes (SNOMED 1631000175102, 716186003) that needed filtering so they wouldn't show as actual allergies.

Building a reliable voice loop — recording audio via MediaRecorder, transcribing with xAI STT, sending to the AI with tool calls, synthesizing the response with xAI TTS, and streaming playback sentence-by-sentence — had a lot of edge cases around timing and state management.

The AI tool call loop needed iteration capping (max 10 rounds) to prevent runaway chains, and we had to inject patient_id into tool arguments automatically since the model doesn't always include it.

Accomplishments that we're proud of

We got real SMART on FHIR OAuth with PKCE working against Epic's sandbox — not a mock, an actual standards-based EHR connection that imports and normalizes 6 FHIR resource types into structured tables.

The full voice pipeline works end to end: speak a question → xAI STT transcribes → Grok reasons with function calling over real patient data → xAI TTS speaks the answer back, sentence by sentence.

The structured symptom charting walks patients through an 8-field clinical intake via natural conversation, saves progress to the database, and generates a shareable chart summary. The body map uses SNOMED CT mapping so clicking a body region connects symptoms to standardized medical codes.

Family multi-profile support lets multiple patients load from Supabase, switchable in the UI, with the AI aware of all family members so you can ask about anyone by name.

We built two share modes: full records (conditions, meds, labs, encounters, allergies with body map visualization) and chart summary (structured symptom intake), both with token expiry and synthetic data banners.

What we learned

The 21st Century Cures Act created a massive untapped opportunity — FHIR APIs are live at every major US hospital, but there are almost no consumer apps built directly on top of them.

Health data is messy even when it's "standardized." Epic's FHIR output has vendor-specific quirks: medication formats vary, allergy sentinels need filtering, observation categories aren't always consistent.

Voice changes everything for health data access. Patients interact with their records completely differently when they can just ask instead of scrolling through a portal. The suggested prompts ("What medications am I on?") get people started, but the real magic is when they ask follow-up questions naturally.

Function calling is the right architecture for this. Letting the AI decide which tools to call (query meds vs. start a charting session) based on natural language makes the two modes feel seamless rather than forcing mode switches.

The biggest competitors (ChatGPT Health, Microsoft Copilot Health) just entered this space in early 2026, which validates the idea — but none of them have a family-first, voice-first, symptom-charting approach.

What's next for ChartTalk

Add Oracle Health (Cerner) and other EHR systems — same FHIR R4 standard, just new OAuth endpoints. Implement recipient identity verification on share links (email/SMS codes) for HIPAA-compliant production access. Build real-time chart updates so families get notified when new labs or clinical notes are added to a connected record. Cross-reference symptom charts with existing records — when a patient reports chest pain, surface their cardiac history and relevant medications automatically. Add multi-language voice support, because health literacy gaps are even wider for non-English speakers. And introduce caregiver role permissions so families can control who sees what level of detail across connected profiles.

Built With

• app-router
• css
• epic-fhir-r4
• fhir-r4
• grok-3-mini
• next.js-16
• openai-tts-api
• openai-whisper-api
• postgresql
• react-19
• smart-on-fhir
• smart-on-fhir-oauth-with-pkce
• snomed-ct
• sql
• stt
• supabase
• tts
• typescript
• vercel
• xai-api