CareLoop

Inspiration

The idea started from a simple, uncomfortable fact: most medication-adherence tools are either a text message nobody reads or a check-in call that's really just a chatbot wearing a phone number. Neither one can tell the difference between a patient who's fine and a patient who's minimizing something dangerous. And an LLM alone genuinely can't be trusted to make that call, because it can be talked down. Someone says "I don't know, my chest hurts, but I'm probably being dramatic," and a model tuned to be agreeable might just agree.

We wanted to build the opposite: a system where the layer that decides "this is an emergency" is never the model. It's a fixed, auditable rule that the model is allowed to escalate but never permitted to soften. Everything else, the natural conversation, the follow-up questions, the tone, is where the model earns its keep.

What it does

CareLoop calls a patient at dose time, asks how they're doing, and listens. What happens next depends on a two-tier safety system:

Tier 0 is a deterministic regex layer that catches emergency and crisis language (chest pain, difficulty breathing, suicidal ideation, and dozens of natural/dialect phrasings of each) before the call ever reaches a model. If it fires, the model never sees the transcript and cannot override the response.
Tier 1 is a Gemini-based classifier for everything that isn't an emergency, handling mild, moderate, and severe triage plus normal conversation.

On top of that: a real drug-interaction check (it correctly catches a Coumadin/Aspirin bleeding-risk pair, for example, and discloses it honestly instead of giving medical advice); a live appointment-booking flow where the agent proposes a specific time, negotiates around "I'm busy" by offering alternatives, and only books once the patient actually accepts; and a check-in summary page that shows exactly what the system decided and why, so nothing is a black box.

How we built it

Backend is FastAPI on Vercel serverless functions, with Twilio handling the phone call itself and Amazon Polly for text-to-speech. Gemini Flash Lite drives the conversational "mild turn" dialogue only. Every safety-relevant decision is deterministic Python, not a model call. The frontend is a React SPA (Vite plus Tailwind) that mirrors what happened on each call: medications, flagged interactions, appointments, and a live trace feed of the agent's decision-making.

Challenges we ran into

This is the part we're most honest about, because the real challenges weren't the obvious ones.

We ran an adversarial security review against our own code and it found that our emergency-detection negation guard (the logic meant to stop "no chest pain" from falsely triggering an emergency) had a hole: a hedge phrase like "I don't know, my chest is really crushing" could suppress the match entirely, because the negation window didn't respect clause boundaries. Two throwaway words could silence a real emergency. We rewrote the guard to stop at commas and "but," verified it against dozens of real phrasings, and it's now covered by regression tests that encode the exact bypass we found.

Separately, we found that Vercel's own rewrite rule was silently injecting an extra query parameter into every request, which broke Twilio's request-signature validation for every single call, correctly signed or not. It took building a temporary debug endpoint that echoed back exactly how the server reconstructed the incoming URL to catch it.

We also found a consent-inversion bug where a patient saying "yes, book it, I have no insurance though" was read as a refusal, because a bare "no" anywhere in the sentence overrode an explicit yes. And a scheduling bug where, once every dose for the day passed its confirmation window, the system had nothing left to say for the rest of the day, silently dropping a patient's "yes, I took it."

None of these were found by us assuming things worked. They were found by deliberately trying to break our own product the way a skeptical judge, or a real, unpredictable patient, would.

Accomplishments that we're proud of

The safety floor actually holds up under adversarial testing, not just happy-path testing. We verified live, against production, that a real hedge-phrase attack no longer suppresses an emergency, that a real drug interaction gets disclosed honestly, and that the booking flow can't be tricked into fabricating a confirmation that never happened. We're also proud of the process: multiple independent adversarial passes (security-focused, then a hostile "hackathon judge" pass specifically testing mobile and demo-readiness) that treated "it works on my machine" as the start of the bar, not the finish line. 589 backend tests and 45 frontend tests passing, several of which exist specifically because we found a real bug and refused to ship the fix without a test that would have caught it.

What we learned

A model is a phenomenal conversational partner and a bad final authority on anything safety-critical. The two-tier design isn't a compromise, it's the actual insight. We also learned that the most dangerous bugs in a system like this aren't crashes, they're silent wrong answers: a suppressed emergency, a falsely-read refusal, a call that goes quiet instead of erroring. Those only surface if you go looking for them adversarially, which is why we built the review process to explicitly try to break our own safety claims rather than just demo the happy path.

What's next for CareLoop

Moving off in-memory session state (currently scoped to a single serverless instance) onto a persistent store so state survives a cold start; adding real session authentication instead of an unauthenticated session header; and expanding the Tier 0 rule set and interaction database with clinical input rather than our own hand-written test cases. Inspiration

The idea started from a simple, uncomfortable fact: most medication-adherence tools are either a text message nobody reads or a check-in call that's really just a chatbot wearing a phone number. Neither one can tell the difference between a patient who's fine and a patient who's minimizing something dangerous. And an LLM alone genuinely can't be trusted to make that call, because it can be talked down. Someone says "I don't know, my chest hurts, but I'm probably being dramatic," and a model tuned to be agreeable might just agree.