• 

• 

Refill: Devpost project story

THE PROBLEM: Every month, an adult child managing a parent's specialty medication gets a refill denial letter, has no idea what the plan's early-refill rule actually is, and burns an afternoon on hold to learn something that's really just arithmetic over three known numbers.

THE SOLUTION: Refill asks the questions a pharmacy would ask, runs the eligibility math itself with a deterministic calculator, and hands the caregiver a one-page packet, medication, NDC, the calculator-derived date, a phone script, before anyone picks up the phone.

WHY IT'S INNOVATIVE: The eligibility calculator sits outside the model and holds veto power over it. Gemini proposes a next-eligible date through an ADK tool call; if that date disagrees with the calculator's own arithmetic, the calculator wins, no packet gets issued, and both dates are recorded side by side. This isn't a disclaimer, it's enforced: we deleted the veto on purpose, a wrong-dated packet.pdf reached disk and four tests failed instantly. Restoring the check killed the run again (RED_GREEN.md).

KEY FEATURES:

• A deterministic calculator with real veto power over the model's date, not a caveat in a system prompt
• Firestore memory with honest deletion: forget_fact actually removes a fact, it doesn't just flag it
• A dialogue engine that never asks the same question twice in one run
• A Cloud Scheduler follow-up tick that appends a genuinely time-delayed second log entry after the packet ships
• Bounded authority: Refill never contacts a payer or pharmacy itself, the blast radius of a bug is a PDF

Try it out (no install)

https://refill-collaborative-agent.vercel.app runs the real offline demo, server-side, live: the model's date gets REJECTED with zero packets when it disagrees with the calculator, ACCEPTED with a real packet.pdf when it agrees, a duplicate tick is a no-op, and a second chase asks fewer questions because it remembers the plan. Zero network, zero GCP, zero API key. Same demo_local.py loop the README walks through.

The three numbers nobody puts in one place

Picture the adult child managing a parent's specialty medication. A denial letter shows up almost every month, not because anything is actually wrong, just because the pharmacy's system doesn't know this plan allows an early refill under specific conditions. Someone loses an afternoon on hold hunting for the actual rule: how many days early this plan allows, whether this month's letter means what last month's did. Nobody on that call knows what an NDC number is, or why it's printed on the letter, or why it matters.

The answer was never hidden. It's arithmetic: last fill date, plus days supply, minus however many days early the plan allows. Three numbers. The hard part was never the math, it's that nobody assembles the three numbers in one place before picking up the phone. We built the thing that does that arithmetic itself, out loud, and remembers the plan's quirks so next month it already knows this plan allows 2 days early instead of 0, and doesn't ask again.

What Refill actually does

A caregiver types in what's on the denial letter, or the parser pulls it out automatically: agent/denial_letter.py extracts NDC, plan, denial reason, last fill date, and days supply straight from the text, no model call. Whatever the letter didn't say, a deterministic dialogue engine (agent/dialogue.py) tracks as missing and asks for it, in a fixed order, so the same run never repeats a question.

Once plan, days supply, and last fill date are known, an ADK LlmAgent running Gemini 2.5 Flash proposes a next-eligible date through a tool call, propose_next_eligible_date. That tool doesn't take the model's word for it. It runs the real calculator itself:

next_eligible = last_fill_date + days_supply - allowed_early_days(plan)

and checks its own answer against whatever the model just said. Agree, and the packet gets written. Disagree, and the run is rejected, zero artifacts, full stop. Both dates land in the run record side by side, so a disagreement is visible instead of quietly overwritten. The model's system prompt tells it plainly: it doesn't get to argue past a DISAGREE or try a second guess. The calculator's arithmetic is final, not a suggestion.

Facts the family already gave once, which plan, how many days the last fill covered, get written to Firestore and read back automatically next month, so the second chase for the same plan is measurably shorter, fewer missing fields, fewer questions, not a vibe. Any stored fact can be corrected or deleted outright: forget_fact actually removes it from future retrieval instead of leaving a stale flag behind. A separate Cloud Scheduler tick checks chases that already shipped a packet and appends a real, time-delayed follow-up line to the log, the kind of thing that has to outlive the original request because it genuinely hasn't happened yet when that request finishes.

Refill never calls a payer or a pharmacy on anyone's behalf. It hands the caregiver a one-page PDF, medication, NDC, plan, the calculator-derived date, an exact phone script, then gets out of the way. The blast radius of a bug here is a wrong PDF, not a wrong claim filed somewhere on a family's behalf.

Architecture

The calculator sits outside the model, a plain Python function with its own unit tests: leap years, a zero-early-day plan, a missing days_supply that raises instead of silently defaulting. The ADK tool wraps the calculator, not the other way around, so the veto can't be talked out of its answer.

Building a calculator the model can't outvote

The dialogue lives in an ADK LlmAgent (agent/refill_agent.py) with the calculator bound in as a FunctionTool. The first deploy attempt used gemini-3.5-flash and hit a hard 404, that model doesn't exist as a Vertex AI publisher model. Switched to gemini-2.5-flash, confirmed live.

The sharpest fix in this project's last stretch was catching a gap between "the agent exists" and "the agent runs on the path a judge would actually hit." A judge-panel pass on Aug 31 read job/main.py after a live Cloud Run deploy and caught it: chase mode was reading REFILL_MODEL_CLAIMED_DATE straight from an environment variable into the validator and never touching agent/refill_agent.py's LlmAgent at all. The calculator was real. The dialogue agent that was supposed to feed it a proposal was decorative, imported, unit-tested, never invoked where it mattered. That's the entire difference between an agent doing something and an agent existing in a repo.

We fixed it by making job/main.py's chase mode call agent/run_chat.py::run_single_turn for one turn through the same InMemoryRunner the interactive CLI already used, so the job entrypoint and the CLI share one runner instead of the job reimplementing its own. Chase mode now requires a live GOOGLE_API_KEY or GEMINI_API_KEY and raises loudly with no key rather than fabricating a conversation, matching the project's own rule against ever guessing. Six new tests in tests/test_job_main.py assert the ADK invocation actually exists in job/main.py's source (inspect.getsource, so reverting that wiring by accident fails the suite, not just the demo), that a missing key raises SystemExit naming the exact env var, and that the model's proposed date really reaches the tool closure and comes back out through _run_dialogue_agent.

Every ADK session stays short, on purpose. Two real upstream adk-python issues drove that call: a 100-event session cap with no pagination (#3559, #3613) and an _init_session replay that could trip a 429 on long sessions (#5714). Both are closed now, not live bugs we're fighting today, but they're why durable state, the plan's remembered facts, lives in Firestore and not in ADK session history.

On memory: add_session_to_memory kicks off an async extraction job and returns before it finishes, so reading facts back right after writing them can miss what you just wrote. Instead of building a wait loop around that race, we skipped the API entirely. memory/profile.py writes facts to Firestore directly and synchronously, remember_fact only returns after the write lands, so there's no extraction step left to race. A memory feature that looks broken because a background job hasn't finished yet is worse than no memory feature at all; the simplest way to not lose that race was to never enter it.

Where it almost went wrong

The honest one is the gap between "the agent exists" and "the agent runs on the path a judge would actually invoke," above. It's an easy trap: everything is unit-tested, the code reads correctly, and none of that tells you whether job/main.py's main() actually calls it. We only caught it by reading the deployed job's source after running it live, not by trusting the test suite's green checkmarks.

The other real one was proving the validator's veto has actual teeth, not just a comment describing intent. We edited validator/refill_validator.py to bypass the disagreement check, returning passed=True even on a mismatch, and re-ran a chase with a last fill of 2026-01-01, a 30-day supply, and the standard plan (correct calculator answer: 2026-01-29). With the veto disabled, a real packet.pdf got written carrying the wrong date, 2026-02-15, the exact "confident wrong date" failure mode this entire project exists to prevent. Four tests failed immediately, the ones that specifically assert the veto's behavior. Restoring the check made the same disagreement produce zero artifacts again. That break-and-restore is recorded in RED_GREEN.md, because a validator you haven't watched fail is a validator you're just trusting.

What's actually deployed

74 offline tests pass: the calculator, the validator, the letter parser, the dialogue state machine, the ADK agent's tool closure, Firestore-backed memory including deletion and correction, the packet writer, the idempotent job tick, all zero network calls, zero GCP credentials required. The one live-model call in the whole suite is stubbed; everything downstream of it, the tool the model calls, the arithmetic, the artifact writing, is real code exercised directly. No mocks standing in for the logic that actually decides anything.

We deployed for real: two Cloud Run Jobs built from one image, a real Firestore-backed idempotency claim, a real packet.pdf and log.jsonl written to a real GCS bucket, then both the accepted path and the rejected path exercised against it before tearing the whole thing down. Fixed a real deploy bug along the way, infra/deploy_refill.sh was building with the monorepo root as Docker context when the Dockerfile's COPY paths assume projects/refill/ as the build context.

The real lesson

The calculator is what turns this from a chat assistant into something a caregiver could actually trust with her mother's medication. A model asserting a date on its own, however fluent, is exactly the failure mode that sent us to the phone every month in the first place: a confident answer from something that sounds authoritative and isn't checking its own math. Watching the RED test actually produce a wrong PDF, not just reasoning that it probably would, is what convinced us the veto was load-bearing, not decorative.

The other lesson was about memory. Giving the caregiver an honest deletion path, forget_fact that actually removes a fact from future retrieval, not a soft flag nobody reads, mattered more than any personalization feature we could have bolted on. A memory that can't be corrected is worse than no memory, because it's a wrong fact the system now insists on.

What's next for Refill

Wire agent/denial_letter.py's parser directly into an upload flow so a caregiver can photograph the letter instead of typing fields in. Re-verify the newly-wired ADK chase path against a fresh live Cloud Run deploy, the current live deploy predates this fix; the wiring is tested locally with the model call stubbed but not yet re-proven end to end on GCP. And extend the plan-quirk memory past early-refill days to the other recurring payer-specific facts caregivers learn the hard way, like which payers want the days-supply field spelled out verbatim on every call.

Technologies used

Gemini 2.5 Flash, Google Agent Development Kit (ADK), Cloud Run Jobs, Cloud Scheduler, Firestore, Cloud Storage, OpenTelemetry, Python, reportlab.

Other data sources

None. No external medical or payer data source, no real patient data. The sample denial letter in fixtures/ is synthetic and marked as such. Plan rules are explicit inputs, not scraped from anywhere, because the whole point is that the arithmetic is auditable.

Built With

• adk
• cloud-run
• cloud-scheduler
• firestore
• google
• google-cloud
• python
• reportlab