Between Visits

An AI voice agent calls cancer-trial patients between visits, in English or Spanish, at their pace. Rules grade symptoms, a nurse is paged in seconds, and sponsors see which sites to check.

Comment

Inspiration

Cancer-trial patients see their study team every few weeks, but side effects start in between and surface at the next visit, from memory. A Regeneron mentor told us the hardest patients to reach and keep are people who don't speak English and people who move slowly: exactly the people that English-only staff, apps and fast voice bots fail. On the sponsor side, today's AE key risk indicators compare one site's AE log with other sites' AE logs, so they can't tell a site with healthier patients from a site that misses symptoms.

What it does

Between Visits phones trial patients 1, 3 and 7 days after each dose, in English or Spanish, at standard or extra-time pace.

  • Discloses it is an AI and offers a human in the first sentence, every call.
  • Asks plain-language symptom questions using approved wording only (our own items; registered studies can load NCI's PRO-CTCAE locally).
  • The LLM listens, rules grade, humans decide: answers become enums and numbers, fixed CTCAE v5 rules give a provisional grade, and a fixed red-flag table decides escalation.
  • Urgent findings page the on-call nurse in seconds (push with an Acknowledge button).
  • After the call: draft AE and medication entries with the patient's verbatim words, timestamp and source, labeled AI translations, coordinator review, investigator seriousness, and a sponsor serious-event handoff with a 24-hour clock.

Site Signal (sponsor side): for each site it compares the site's own AE log with what that site's patients told the agent, adjusted for symptom mix. The output is capture discordance: a reason to look, not a finding. Every flag drills down to patients' own words; a CRA approves every templated, non-leading data-completeness query; the site answers in a mock inbox; every step is audited in a hash-chained log.

How we built it

  • Voice: ElevenLabs agents (4 variants: EN/ES × standard/extra time) on Claude, config-as-code with a deployed-config hash check in CI.
  • Backend: FastAPI + SQLModel. A deterministic interview engine and rules engine; the LLM never decides a grade, red flag or escalation. Post-call pipeline on the Anthropic API with evidence spans.
  • Frontend: Next.js + TypeScript dashboard with role views (coordinator, investigator, sponsor, CRA). Every AI string is labeled; every Site Signal view is badged SIMULATED or MEASURED; no tier is ever green.
  • Statistics (numpy/scipy, no LLM anywhere on that path): leave-one-site-out O/E with cluster-robust variance and additive overdispersion, $$z^*_s = \frac{O_s - E_s}{\sqrt{V_s + \hat\tau^2 E_s^2}},$$ with Look and Watch thresholds calibrated on 2,000 simulated null studies. Compared on identical data with a gsm-style AE-rate KRI and a port of simaerep classic, which reproduces its R regression p-values to a relative error of 3.5e-15.
  • Pre-registered: the analysis plan, parameters and data files were frozen under the git tag sap-v1 before any result existed. The evaluation seed came from that commit plus a public drand beacon round published 10 minutes later. Every evaluation run is logged, and a test checks the order in CI.

Results

Voice agent, text-mode evaluation, 20 simulated patients (Clopper-Pearson 95% CIs):

  • Red-flag recall 8/8 (0.63–1.00); AE recall 1.00 (0.82–1.00); AI disclosure 1.00 (0.83–1.00); approved wording 0.93 (0.89–0.96).
  • 19/20 personas passed (0.75–1.00). The known miss is P19, an escalation false alarm.

Site Signal, SIMULATED 20-site study (Monte Carlo SE in brackets):

  • Calibrated as pre-registered: false Look flags 0.048 per honest site [0.001]; studies with any false Watch flag 0.108 [0.007].
  • Watch / Look power, one planted site:
    • large capture loss (f = 0.90): 0.70 / 0.77
    • 50% loss: 0.22 / 0.51
    • 25% loss: 0.03 / 0.15
  • What it gets wrong, honestly: moderate gaps go mostly unflagged at Watch. On a plain large gap, the simple AE-rate KRI catches more (Look 0.96 vs 0.67 in the full planted mix). Discordance earns its place where rate KRIs fail: it almost never flags the honest site with healthier patients (0.001 vs 0.042 for the rate KRI and 0.234 for simaerep classic), and it is the only indicator that sometimes catches a gap masked by sicker patients (0.10 vs 0.00).

Challenges we ran into

  • Licensing: NCI's PRO-CTCAE wording and MedDRA can't go in an open repo, so we wrote our own items, with a local plug-in for registered studies.
  • Telephony: outbound calls need carrier compliance approval, so calls run in the browser and the nurse page uses push.
  • Paging changes the statistics: every fever and grade 3 episode pages the nurse (so the site learns of it from us). Those episodes form a separate stratum, and the analysis rests mostly on moderate grade 2 symptoms. Our pre-registered planning rules came up short, and we report that as is.
  • A real pre-registration under hackathon time: an independent statistical review found 13 issues across 4 rounds, all fixed before the tag.

Accomplishments we're proud of

  • Safety rules enforced by code and tests, not prompts: 1,045 passing tests, 29 traced requirements and a CI secret scan.
  • A statistical claim we can't quietly tune: tag, public beacon, run log.
  • Honest limits: WHAT_IT_GETS_WRONG.md, plus 27 GitHub issues for what we didn't build.

What we learned

Plain-language voice reaches people that forms don't, but only if the AI never improvises clinical words. For monitoring, the patient's own report is a reference, not the truth: capture discordance includes legitimate clinical judgment, so the product asks questions and never accuses.

What's next

  • Retrospective validation on trials with paired patient- and clinician-reported (PRO-CTCAE/CTCAE) data: the only non-circular test.
  • A one-site pilot, more languages with professionally translated wording, and EDC/FHIR integration.

Synthetic data only. Not a medical device. The study drug is referred to by class (anti-PD-1).

Built With

Share this project:

Updates

Submission history