AfterVisit - Closing The Loop

Your personal friendly voice AI assistant that has real-time access to your EHR records and helps you through the post-visit workflow

Comment

Inspiration

Patients leave the clinic overwhelmed. The research is brutal: people remember only 49% of what their doctor said, 31% of newly prescribed diabetes medications are never filled, and the patients with the worst follow-through face 45% higher mortality and are 58% more likely to be hospitalized. The moment you walk out the door, you're alone with a wall of jargon, a prescription you don't understand, and a follow-up you'll probably forget. The system gives you nothing between visits. We wanted to build the companion that closes that loop — something that feels like calling a friend who happens to be a nurse.

What it does

Aftervisit is a post-visit AI patient companion. Right after your appointment, you get a phone call from Remy, your friendly AI care guide. In one natural, low-latency voice conversation, Remy:

  • Greets you and asks what you want to make sense of — it never dumps information on you.
  • Translates the jargon into plain English, and as it explains, it highlights the exact lines in your real records so you can see what it's talking about.
  • Answers your questions grounded in your chart (e.g., "how much is all of this going to cost me?" → pulls your real estimated costs and breaks them down).
  • Takes action with your consent — when you say yes, Remy spawns named sub-agents to do the work (text your pharmacy, book a follow-up, find an in-network eye doctor, flag a record error). You watch them work in real time as glowing cards in the corner. Nothing happens without your okay.
  • Connects you to community — when you say you're scared and want people to talk to, Remy warmly offers to connect you with an opted-in peer nearby with the same condition.

Everything is voice-first, grounded in your own data, and consent-first by design — a companion, not an autopilot.

How we built it

  • Frontend: Next.js (App Router) + React + TypeScript + Tailwind, with Zustand for state and Framer Motion for the animated, "Clicky-style" sub-agent cards.
  • Voice (xAI Grok): The live call runs on the Grok realtime voice agent (grok-voice-think-fast-1.1 over wss://api.x.ai/v1/realtime). Because a browser can't attach an auth header to a WebSocket, we built a small Node ws proxy that holds the key and injects Remy's persona + the patient's full EHR corpus into the session so every answer is grounded. Mic audio streams as PCM16 @ 24 kHz via an AudioWorklet, the model's voice plays back with barge-in (you can interrupt it mid-sentence), and live transcripts stream into the chat.
  • Reasoning brain (Claude): Typed questions are answered by a local Claude CLI subprocess (claude -p … --output-format json) reasoning over the synthetic /EHR corpus — no cloud API key needed. It returns {answer, scope, citations}, so the records panel can open and highlight the exact cited line.
  • Grounding: A generator script builds a realistic dummy EHR (after-visit note, A1c labs, med list, estimated costs, problem list, prior visits) as real PDFs plus structured blocks the UI renders and highlights.
  • Agentic actions: Each consented action spawns a named sub-agent (Jack = records, Pax = pharmacy, Cal = scheduling, Mira = community, Quinn = records office), each a distinct color with a subtle glow, and lands an animated "✓ done" card under "With your OK."

Challenges we ran into

  • Realtime voice over WebSockets in the browser — you can't set an Authorization header on a WS, so we had to build a relay proxy and learn the OpenAI-realtime-style event protocol (server VAD turn detection, audio deltas, transcription events) the hard way.
  • One voice, many agents — making it feel like a team without ever having two voices talk over each other. Only Remy ever speaks; the sub-agents are visible workers, never voices.
  • Spawning a local Claude subprocess that wouldn't hang — the CLI waited forever on an open stdin pipe until we set stdio: ["ignore","pipe","pipe"].
  • Consent-first flow detection — distinguishing "I'm looking for community" (Remy should offer first) from "yes, connect me" (now spawn the agent) so actions only fire after the patient agrees.
  • Making it unbreakable for a live demo — a fully scripted fallback path takes over instantly if the realtime connection ever drops, so the demo can never die on stage.

Accomplishments that we're proud of

  • A genuinely seamless, low-latency voice conversation that greets, explains, highlights records, answers cost questions, and takes consented actions — all in one natural call.
  • The records panel that highlights the exact line Remy is talking about, in real time.
  • Consent-first agentic actions with live, glowing sub-agent cards — visible agency without ever losing patient control.
  • A real, warm community-connection moment that lands the emotional core of the product.
  • Three sponsor tools woven into one coherent story: Grok Voice for the conversation, local Claude for grounded reasoning, v0 + Cursor for the build.

What we learned

  • Patient agency is a design problem as much as an AI problem — the magic is in asking first and showing your work, not in doing more automatically.
  • Grounding + citation highlighting is what turns "an AI chatbot" into "something I can actually trust with my health."
  • Realtime voice is a different beast from request/response TTS — barge-in and turn-taking are what make it feel human.

What's next for Aftervisit

  • Real EHR integration (FHIR) instead of synthetic data, behind proper auth and consent.
  • True peer-matching with a real, opted-in community.
  • Live tool execution (real pharmacy/scheduling APIs) behind a verified-action audit trail.
  • Multilingual support and accessibility-first voice for the patients who need this most.

Built With

Share this project:

Updates