Unsay

An AI medication-safety agent that goes back and un-says what it told you. When the FDA escalates a recall, it finds every patient it already reassured and corrects them by name, in seconds.

Comment

Inspiration

A recall is scoped to a lot number. If you cannot resolve a lot number to a person, you either warn everybody or you warn nobody, and in practice the industry warns nobody.

That is not my opinion. A 2024 study at an academic medical center, "Automating Individualized Notification of Drug Recalls to Patients", built a system to scan FDA recalls nightly and message affected patients through their EHR portal. It hit two walls. Most recalls are Class II, so notification "stops with wholesalers and pharmacies rather than reaching patients". And critically: "it was not possible to trace a medication prescription from the EHR to specific lot numbers dispensed to that patient by a community pharmacy".

Separately, the named Day-2 failure of AI agent memory in 2026 is stale context: similarity to a stored memory does not prove that memory is still true. An agent retrieves a fact that was correct when written and answers as if it were correct now. For most agents that is embarrassing. In a pharmacy it is a Class I recall, which the FDA defines as a reasonable probability of serious adverse health consequences or death.

Those are the same problem. Both are about a fact that has changed and a system that does not know which of its past answers depended on it.

What it does

Unsay answers medication-safety questions from live openFDA recall and labelling data. That part is ordinary.

The part that matters happens later. When the FDA escalates a recall, Unsay finds every answer it has already given that leaned on the old version of that claim, re-decides each one against what is known now, and drafts a correction addressed to the specific person, for a pharmacist to approve.

In the hosted demo that is twelve patients dispensed one real recalled lot. The sweep examines all twelve, reverses nine, and leaves three alone because they had already been told to stop. Nobody receives a message whose answer did not actually change.

How I built it

CockroachDB is the memory, and the schema is the whole idea.

Every safety claim is bitemporal. valid_from and valid_to record when the claim is true in the world; asserted_at and retracted_at record when this system believed it. Keeping those apart is what separates "the drug became dangerous in March" from "we found out in July", which is the difference between an unlucky answer and a negligent one. A claim is never updated in place: the believed version is retracted and a new one asserted beside it, in one transaction.

Every answer records exactly which claim versions it read, written in the same transaction as the answer itself. There is no code path that stores an answer without its provenance, because an answer whose provenance was lost can never be repaired.

Those two things together make the flagship query possible:

SELECT d.decision_id, d.answer, d.verdict
  FROM decision d
  JOIN decision_read dr ON dr.decision_id = d.decision_id
  JOIN fact stale       ON stale.fact_key = dr.fact_key
                       AND stale.version  = dr.fact_version
 WHERE d.status = 'standing'
   AND dr.load_bearing
   AND stale.retracted_at IS NOT NULL;

Every answer still standing that leaned on a version of a claim we no longer believe. A vector database cannot express this. It has no notion that a memory has versions, and no record of which version an answer consumed.

Retrieval is CockroachDB's distributed vector index over Titan V2 embeddings, prefixed on a stored computed believed column so a top-K search never spends part of its budget on claims already retracted. Claude Sonnet 4.5 on Amazon Bedrock does the reasoning and names its own citations, which is what makes the sweep possible: an answer can only be invalidated later by evidence the model itself said it used. The hosted demo runs on AWS Lambda against a CockroachDB Cloud cluster; the multi-region resilience work runs on a 9-node, 3-region cluster under SURVIVE REGION FAILURE.

Challenges I ran into

The obvious design was wrong, and CockroachDB's own documentation told me so. Storing a read timestamp and replaying with AS OF SYSTEM TIME is elegant and needs no extra tables, and it works until the garbage collector passes the timestamp you saved. The default window is four hours. Twenty-five is the largest value Cockroach Labs regularly tests. Their docs say plainly that gc.ttlseconds "is not meant to be a solution for long-term retention of history; for that you should handle versioning in the schema design at the application layer". So the durable mechanism is the bitemporal schema, and MVCC time-travel is a fast path inside the window. scripts/expiry.py demonstrates the difference: at 45 days the bitemporal route answers exactly and the MVCC route fails on the replica GC threshold, while inside the window both agree.

Making supersession actually fire took three attempts. Extraction silently dropped whole sessions when the model answered conversationally instead of returning JSON, so a fact that should have superseded another simply never landed, and the failure looked like a memory bug. Then extraction turned out to be memory-blind: each session invented its own key for the same attribute, so nothing ever superseded anything. Even after showing the model the existing keys, it emitted recent_trip_destination_paris against a known recent_trip_destination, encoding the value in the key. A qualifier appended to a known attribute is a new value for that attribute, not a new attribute, so that is now enforced in code rather than asked for in a prompt.

Deploying surfaced three more that testing never would. The Lambda bundle omitted a transitive dependency that the local environment happened to have. Function URLs created after October 2025 need both InvokeFunctionUrl and InvokeFunction even with AuthType NONE, which every older guide gets wrong and which returns a 403 against a policy that reads as entirely correct. And the status endpoint treated "no probe configured for this region" as "region down", so the deployed demo reported its own healthy region as failed.

Accomplishments that I am proud of

The demo survives repeat visitors. The sweep is destructive by design, so the first person to run it would have left every judge afterwards clicking a button that does nothing for four weeks of judging. It now detects a spent scenario on page load and restores itself.

The cost of a public unauthenticated demo is bounded. A sweep re-decided twelve answers to the same question and paid for twelve identical replies; identical questions now collapse to one call. The opening answer is cached, so a reset spends nothing. Then a hard ceiling of 200 model calls a day and 40 per visitor per hour, counted in the database rather than process memory because Lambda has many short-lived processes.

The bitemporal invariants hold under contention: 64 concurrent writers with 16 racing on each of four claims, zero failures, version chains dense with no gaps or duplicates, exactly one believed version per claim.

And the honest reporting. The benchmark I ran came out badly and it is in the README with its diagnosis rather than deleted.

What I learned

Running the real thing finds what testing cannot. Every significant defect in this project was found by deploying it or by trying to record it, not by a test: the dependency that only existed locally, the 403 with a correct-looking policy, the region that reported itself down, the warning that printed 64 times and buried its own result, twelve corrections with identical text.

I also learned to stop when a number will not come. I measured Unsay on LongMemEval's hard split with the official GPT-4o judge and scored 20.5%. The figures usually quoted for the same sub-task are 63.8% for Zep and 49.0% for Mem0, and those deserve a caveat I did not give them at first: they come from a third-party comparison rather than from Zep's own paper, which reports "improvements up to 18.5%" and does not state 63.8% anywhere I could find. So treat them as indicative of the gap rather than as a matched baseline. What is not in doubt is the direction. The diagnosis is specific: 25 of 31 wrong answers were "I don't know", with a median 125 facts stored per instance against a retrieval window of 10. Widening the window to 40 moved a fixed sample from 25% to 33.3%, which is one extra correct answer and noise at that size. That is a retrieval problem, it is unsolved here, and it is reported rather than buried, because omitting a measurement while shipping the harness that produced it would be worse.

What's next for Unsay

Reconciling lot extraction against real pharmacy dispensing records. Lot numbers currently come from regex over openFDA free text, and recalls with no parsable lot fall back to drug-level scope, which over-notifies rather than under-notifies. That is the safe direction to fail, but it is not the right one.

Then the retrieval problem the benchmark exposed, properly: adaptive k, or recall-oriented reranking, measured the same way and reported the same way.

Built With

  • amazon-bedrock
  • amazon-titan
  • aws-lambda
  • boto3
  • ccloud
  • claude
  • cockroachdb
  • cockroachdb-cloud
  • docker
  • fastapi
  • html
  • javascript
  • mangum
  • mcp
  • openfda
  • psycopg3
  • pytest
  • python
  • sql
  • vector-search
Share this project:

Updates