IntakeAI Brain — Zero-Trust Patient Intake

WE SOMEHOW FIGURED OUT TO RUN THIS WITH LOCAL SETUP AS TWILIO WAS NOT WORKING. WE CHECKED WITH ANAND FROM ARYA HEALTH AND VERIFIED THAT THIS IS OKAY CAUSE WE WERE FACING ISSUES WITH TWILIO. WE HAVE UPLOADED A SMALL DEMO WHICH SHOWS MOST OF OUR FEATURES WORKING, GIVEN THE TIME CONSTRAINTS

Inspiration

Home health agencies lose a third of their patients before care even begins — not because they can't provide the care, but because nobody answered the phone. When a hospital discharge planner calls at 7 PM to place a 72-year-old hip replacement patient, she gets voicemail. She doesn't wait — she calls the next agency on her list. Meanwhile, a 50-page fax referral sits in a queue for 70 minutes while a coordinator manually reads through it, types data into the EMR, and chases missing documents.

The numbers are staggering: referral conversion rates have dropped to 64%, agencies miss 20% of referrals that arrive outside business hours, and the industry collectively loses $200-500 million annually to referral leakage. The bottleneck isn't clinical — it's administrative. Speed wins patients, and agencies are too slow.

We built IntakeAI for Arya Health, who has automated the caregiver side of home health with 5 AI agents (staffing, compliance, onboarding, payroll, engagement) but has nothing handling the patient intake side. The Intake Agent is their publicly stated next product — announced during their Series A — and it hasn't shipped yet. We decided to build it.

What it does

IntakeAI is an AI-powered patient intake agent that ensures a home health agency never loses a patient because nobody picked up the phone.

Answers every call 24/7. A discharge planner calls and gets a structured clinical intake conversation. The agent extracts patient details in real-time — diagnosis, insurance, zip code — checks eligibility against the agency's actual capabilities (service areas, accepted insurance, caregiver availability, certification requirements), and gives an answer on the spot: "We can accept this referral. We have a nurse available within 48 hours. Send us the face-to-face documentation." Three minutes, done.

Handles family calls with empathy. A worried daughter calls at midnight. The agent speaks compassionately, collects what she knows, gives a preliminary answer, and schedules a coordinator follow-up for the morning. She doesn't have to call five agencies hoping someone picks up.

Reads faxes automatically. A 50-page referral packet goes through a 7-layer extraction pipeline with an agentic validation-correction loop. The system classifies each page, extracts structured data, validates ICD-10 codes and medication dosages against reference databases, catches OCR errors, and flags what's missing — all in minutes instead of 70.

Follows up automatically. When documentation is missing, the agent makes outbound calls to collect it. If the referring provider doesn't answer, it leaves a voicemail and sends an SMS follow-up. It retries on a schedule until the gap is closed or a human coordinator is notified.

Shows everything on a dashboard. A real-time intake pipeline view shows every referral's status, extracted data with confidence scores (green/yellow/red), gap tracking, call transcripts, and caregiver match results.

How we built it

Architecture: A LangGraph orchestrator coordinates four sub-agents — the Voice Agent (Twilio ConversationRelay for phone calls), the Document Pipeline (7-layer extraction), the Eligibility Agent (Neo4j + PostgreSQL traversal), and the Follow-up Agent (SMS, outbound calls, retries).

Voice: Twilio ConversationRelay handles the telephony — Deepgram for speech-to-text, Google for text-to-speech, with a WebSocket connection to our backend. Gemini Flash powers every conversation, returning both a natural language response and structured extracted data in a single API call (Option A combined extraction). A guardrail service checks every outgoing message against blocked patterns before it reaches the caller.

Document Pipeline: Fax PDFs go through ingestion, page classification, dual-path extraction (regex for clean digital PDFs, Gemini vision for scanned/handwritten pages), an agentic review loop with three agents (Validation, Correction, Cross-Reference), completeness checking, and confidence scoring. The Correction Agent gets feedback when its first attempt isn't confident enough and retries with enriched context — up to 2 attempts per field.

Knowledge Graph: Neo4j holds the clinical intelligence — 30 ICD-10 diagnoses mapped to required service types, service types mapped to required certifications, 9 insurance plans with coverage rules (prior auth requirements, required documents, visit limits), and 20 medications with dosage ranges and contraindications. The Eligibility Agent traverses this graph to answer: "does a valid path exist from this patient's diagnosis through required certifications to an available caregiver, with valid insurance coverage?"

Databases: PostgreSQL for operational data (intake records, caregiver roster, insurance contracts), Neo4j for the knowledge graph, Redis for pipeline state and real-time call state caching.

Frontend: React + Tailwind dashboard with three views — pipeline overview, referral detail with confidence-scored extraction results, and a live call monitor that shows data populating in real-time as the caller speaks.

Stack: FastAPI, LangGraph, Gemini Flash, Twilio ConversationRelay, Neo4j, PostgreSQL, Redis, React, Tailwind CSS.

Challenges we ran into

Mid-call latency. The caller is on the phone — they can't wait 10 seconds of silence while we check eligibility. The full loop (extract data → query Neo4j → query PostgreSQL → inject result into Gemini → generate response → guardrail check → send to Twilio TTS) had to complete in under 3-4 seconds. We used filler phrases ("Let me check our availability...") while the eligibility check runs, and cached results in Redis to avoid redundant queries.

Gemini JSON consistency. We rely on Gemini returning structured JSON with specific keys (response, extracted, ready_for_eligibility) on every turn. It doesn't always cooperate. We forced JSON output via the response_mime_type parameter and built defensive parsing — if JSON fails, the conversation continues with just the spoken response, and extraction is skipped for that turn. The call never breaks because extraction failed.

Guardrail enforcement without killing the conversation. The agent can't say "you're admitted" before eligibility confirms, can't give medical advice, can't read back full insurance IDs. But blocking a response mid-conversation creates an awkward pause. We built a feedback loop — when a guardrail fires, the violation is fed back to Gemini with instructions to rephrase, and the corrected response is sent to the caller. The caller never hears the blocked version.

Medical document chaos. Real referral fax packets are a mess — handwritten physician notes, smudged insurance cards, mixed document types in one PDF, inconsistent formats across hospitals. A single OCR error in an insurance member ID means a denied claim. The agentic review loop was essential — the Validation Agent catches the error, the Correction Agent reasons about what the correct value likely is using the source image and reference data, and confidence scoring ensures uncertain values are never silently accepted.

Conversation mode detection. The same phone number gets calls from discharge planners (clinical, fast), worried family members (emotional, need hand-holding), and patients (confused, elderly). The agent has to identify the caller type in the first turn and completely shift its personality, vocabulary, and pace. Getting the system prompt switching right — without re-asking "are you a provider?" after every misclassification — was tricky.

Accomplishments that we're proud of

A real phone conversation that actually works. You call the number, talk naturally, and the agent extracts your data, checks eligibility against a real knowledge graph, and gives you a grounded answer — not a canned response. The mid-call eligibility check, where the agent pauses, queries the database, and comes back with a specific answer about coverage and caregiver availability, is the moment that feels like magic.

The agentic document extraction loop. The pipeline doesn't just do dumb OCR. When a field fails validation, the Correction Agent re-examines the source image with the validation error as context and tries again. When data conflicts across pages, the Cross-Reference Agent reasons about which value is correct. Every field gets a confidence score, and the system honestly tells you what it's not sure about.

The confidence-scored dashboard. Watching the Live Call Monitor during a demo — fields appearing in real-time with green/yellow/red confidence dots as the caller speaks — makes the invisible work visible. Judges can see the system thinking.

Building Arya Health's next product. Everything we built maps directly to a real product gap at a real company. The 5 caregiver-facing agents exist. The patient-facing intake agent doesn't. We built it.

What we learned

Healthcare data is a graph, not a table. The eligibility check — "hip replacement requires skilled nursing which needs an RN with orthopedic certification who covers zip 11201 and is available Tuesday, covered by Medicare Part A which requires face-to-face documentation" — is a 6-hop traversal. In SQL that's a nightmare of joins. In Neo4j it's one clean query. The data model choice made the Eligibility Agent possible.

The phone is still king in healthcare. We came in thinking fax processing was the hard problem. It is technically complex, but the phone call is where patients are won or lost. A discharge planner decides in 3 minutes — whoever answers and gives a clear yes/no gets the patient. The voice agent isn't a nice-to-have, it's the core product.

LLMs need guardrails, not just prompts. Telling Gemini "don't give medical advice" in the system prompt works 95% of the time. The other 5% is where a worried daughter asks "will my mom be okay?" and Gemini empathetically says something that crosses the line. The code-level guardrail service — regex pattern matching on every outgoing message, before it reaches the caller — catches what prompt engineering misses.

Confidence scoring changes everything. The difference between "we extracted this data" and "we extracted this data and we're 87% confident in the ICD code but only 43% confident in the insurance member ID" is the difference between a system that silently gets things wrong and one that honestly tells you where to focus human attention.

What's next for IntakeAI

EHR integration. Today the agent creates intake records in our database. In production, it would write directly to WellSky, MatrixCare, Axxess, or KanTime — the EMR systems agencies actually use. Arya already has these integrations for their other agents.

Real fax receiving. We process uploaded PDFs, but a production system would connect to cloud fax services (eFax, Consensus) to automatically ingest referrals the moment they arrive.

Handoff to Arya's Staffing Agent. When IntakeAI accepts a patient and matches a caregiver, the next step is scheduling the first visit — that's what Arya's Staffing Agent does. Connecting the intake output to the staffing input would close the entire referral-to-first-visit loop.

Multi-language support. 3-4 of our sample caregivers speak Spanish, and many home health patients in NYC are Spanish-speaking. Adding Spanish conversation support would meaningfully expand which patients the agent can serve.

Prior authorization automation. When the eligibility check identifies that prior auth is required (Medicare Advantage, some commercial plans), the next step is submitting clinical documentation to the payer and tracking approval. That's a separate agent — the Prior Auth Agent — that could extend IntakeAI's workflow.