Palimpsest

Agent memory that forgets. A memory is a claim with a lifecycle, not a chunk of text - so it can be contradicted, superseded, and killed. Naive RAG served 3 dead facts as truth. Palimpsest served 0.

Comment

Inspiration

Ask any AI agent with memory what port your dev server runs on. It answers instantly, confidently - and if the port ever changed, wrong.

Not because retrieval failed. Because retrieval worked exactly as designed.

Every memory system shipping today stores text, embeds it, and retrieves whatever is most similar to your question. Watch what that does to two claims that flatly contradict each other:

"We decided to use Postgres."    vs   "We decided NOT to use Postgres."      0.93
"The API listens on port 3000."  vs   "Port 3000 is where the API listens."  0.91

The contradiction scores higher than the paraphrase. To a vector store, "we decided to use Postgres" and "we decided not to use Postgres" are more alike than two ways of saying the same true thing - because embeddings capture topic, not truth. The single word that reverses the entire meaning barely moves the number.

You cannot fix this with a threshold: any cutoff that keeps the paraphrase keeps the contradiction. So the retriever hands the model the live fact and the dead one, ranked side by side, with no way to tell them apart - and the model serves whichever won the cosine coin-flip.

Why nobody fixes it: memory systems store chunks. A chunk is not a unit of truth - one paragraph holds five facts, three still true and two dead. You cannot delete half a chunk. You cannot edit it, because you don't know which sentence went bad. So the only move left is to append a new chunk and leave the old one there. Every memory system is append-only not for lack of imagination, but because they have no unit small enough to kill.

The MemoryAgent track asks for "timely forgetting of stale info." Most entries will build a remembering machine. The track is asking for a forgetting machine.

What it does

A memory here is not a chunk. It is a claim: one atomic assertion, independently true or false

  • which means it can have a status, which means it can die.

Each claim carries:

  • Provenance - where it came from, and when it became true in the world.
  • Kind - because facts don't rot at one rate. identity has a ten-year half-life; config has thirty days; an event never decays at all, because a thing that happened cannot become false.
  • Confidence - decaying exponentially from birth at the rate its kind implies.
  • History - when a claim dies we keep the body, the killer, the date, and the reason.

Four mechanisms:

  1. Extraction - a transcript is not a memory. Qwen distils it into atomic claims.
  2. Adjudication - a new claim doesn't just get appended. We retrieve what it might collide with and ask Qwen to rule: update, contradiction, refinement, or genuinely new? Cosine finds the candidates; only reasoning can decide which one is dead.
  3. Decay - confidence erodes at a rate set by what kind of fact it is. Low-confidence claims are surfaced for re-verification rather than served as fact.
  4. Verification - claims that assert something checkable get re-checked, and demoted when the world moves on.

Nothing is overwritten. You can always ask: what did you used to believe, and when did you stop?

Try it live: https://palimpsest.zettabyteincorp.com - type a fact that contradicts what it believes and watch the old belief die: struck through, with the reason it was killed, and the claim that replaced it underneath. Real calls to Qwen. Nothing staged.

How we built it

Qwen Cloud, on the DashScope International endpoint:

Role Model
Adjudication - the one call that decides whether a belief lives or dies qwen3.7-plus
Bulk claim extraction - cheap, fast, high volume qwen3.6-flash
Collision retrieval text-embedding-v4

Alibaba Cloud Function Compute 3.0 (Singapore), behind a custom domain with a Let's Encrypt certificate. The deployed function serves the audit view, answers questions from what it currently believes, and accepts new transcripts - so a judge can contradict it in the browser.

TypeScript. SQLite (node:sqlite). No vector database. At a few thousand claims, brute-force cosine is microseconds - and the hard problem here was never retrieval speed. It was deciding which retrieved claims are still true.

There is also an MCP server, so any MCP-capable agent can plug Palimpsest in as its memory layer.

Challenges we ran into

The first benchmark said the idea was worthless. naive RAG 88%, Palimpsest 88%. Identical. Adjudication worked mechanically - four claims correctly superseded - it just didn't matter. That result is still in the repo.

Two flaws, both mine. The fixture leaked the answer: every change announced its own death ("it's on port 4000 now, 3000 was colliding"). Real speech doesn't work like that - you say "we're on 4000 now"; you don't file a death certificate for 3000. The naive memory was retrieving the obituary next to the corpse. And 15 claims with top-5 retrieval hands the model a third of the store - the failure mode we target can't even occur when the disambiguating context lands in the window by accident. Before re-running I pre-registered the condition: if the corrected benchmark still shows no gap, the thesis is wrong and I change the design, not the chart.

Then the measuring instrument betrayed me. The grader marked two systems that returned the identical answer ("Session cookies") one correct and one wrong. Same input, different verdict. An instrument that disagrees with itself cannot certify anything.

Fixing it (majority of three independent votes, with the disagreement rate printed next to every number it produces - now 0 disagreements across 19 questions) exposed something worse. Claim IDs are random UUIDs, minted per run, and they were being interpolated into the adjudication prompt. So two runs over byte-identical inputs built different prompts, missed the cache, and re-sampled the model every time. The benchmark had been quietly re-rolling its own dice - and one of my published numbers was wrong because of it.

Deploying to Function Compute had two traps, neither discoverable from the error message:

  1. node:sqlite needs Node 22+, and every Function Compute runtime stops at Node 20 - managed nodejs20 and the custom.debian10 image alike. A naive deploy crashes on cold start. The fix is to bring your own Node: we vendor Node 24 into the code package (checksum-verified) and exec it from bootstrap. That isn't a hack around the platform - it's precisely what a custom runtime is for. The application deploys unmodified.
  2. Alibaba force-downloads any HTML served from its own domains. Content-Disposition: attachment is added to every text/html response from *.fcapp.run, *.aliyuncs.com and OSS static hosting, in every region. It's an anti-abuse policy with no setting to disable it - your browser downloads the audit view instead of rendering it. The API worked the whole time; only the one artefact a human would look at was broken. A custom domain is the only supported escape.

Accomplishments that we're proud of

The benchmark, and the fact that we believed it when it hurt.

12 sessions of a real-shaped project across three months. Facts change: the database is swapped, the launch slips, the brand colour moves, the PM is replaced. Then we ask what is true now. The baseline is naive RAG - chunk, embed, top-k - given the same extraction, the same embeddings, and the same answering model. The only difference is that one of them can kill a claim.

naive RAG Palimpsest
Facts that changed 36% (4/11) 73% (8/11)
Facts that never changed 88% (7/8) 88% (7/8)
Overall 58% (11/19) 79% (15/19)
Served a DEAD fact 3 0

Twice as accurate on facts that moved - and no worse on the facts that didn't. That second row is the one that could have killed this project: a memory eager enough to forget that it destroys stable facts would be worse than append-only, not better. It's printed as loudly as the row that flatters us.

Naive RAG served three dead facts - "Postgres", "September 1st", "#1E4D8C" - with total confidence, weeks after each one died. Palimpsest served none.

And when we fixed the grader, one of our own published numbers turned out to be wrong. naive RAG serves 3 dead facts, not the 4 we'd reported - one corpse was an artefact of our own non-determinism. The corrected number is less flattering. It's the one in the table.

Reproducible, not just claimed. Every model call is cached and committed:

PALIMPSEST_CACHE_ONLY=1 pnpm bench     # 448 cache hits, 0 misses - no API key, no spend

Clone it, replay it, get bit-identical numbers. PALIMPSEST_CACHE_ONLY=1 makes a cache miss throw rather than quietly hit the API, so a replay cannot silently drift from what's published. We'd rather you checked than trusted us.

What we learned

The benchmark is the product. Anyone can build something that forgets. The hard part is forgetting only what's dead - and the only way to know whether you've done that is to build an honest measuring instrument and then be willing to publish what it says.

Ours told us the idea was worthless. Then it told us the grader was broken. Then it told us one of our numbers was wrong, in our own favour.

Each time, the fix went to the design or to the instrument. Never to the chart.

We also learned that cosine similarity is a shortlisting tool and nothing more. It cannot rule. The entire project is the argument that the decision about what's still true belongs to a reasoning model, not to a distance metric.

What's next for Palimpsest

Fix retrieval, which is now the bottleneck. Three of eleven changed facts still fail, and they're honest failures we report rather than hide: it answers "ams" (the Fly.io region) instead of "Fly.io" because the region claim outranked the platform claim - the correct claim was alive in the store and simply wasn't reached. That's a retrieval failure, not an adjudication failure, which means the next win is reranking (qwen3-rerank), not more reasoning.

Durable storage. The deployed function's filesystem is ephemeral: it ships with a pre-seeded store, and writes made during a session live in /tmp and vanish when the container recycles. Fine for a demo, wrong for a database - so we say so instead of implying durability we don't have. The fix is a managed store behind the same claim interface.

Active verification. Decay currently flags a claim as stale. The next step is for the memory to go and check - re-read the config file, re-query the API - and demote itself before you ever ask.

The build journey

I wrote up how this was actually built - including the two times the benchmark told me I was wrong, and what I changed each time:

https://joyahmed.github.io/palimpsest/

palimpsest (n.) - a manuscript scraped clean and written over, where traces of the earlier text still show through.

Built With

  • agent-memory
  • ai-agents
  • alibaba-cloud
  • benchmarking
  • cosine-similarity
  • dashscope
  • embeddings
  • function-compute
  • html
  • javascript
  • knowledge-graph
  • lets-encrypt
  • llm
  • mcp
  • model-context-protocol
  • node.js
  • prompt-engineering
  • qwen
  • qwen-cloud
  • rag
  • serverless
  • serverless-devs
  • sqlite
  • typescript
  • vector-search
Share this project:

Updates