Palimpsest

Agent memory that can be wrong, and knows it.

Comment

Inspiration

Almost every agent memory system treats memory as an append-only log of facts. That model is wrong in a specific, expensive way.

An agent's memory is not a log. It is a belief state, and belief states go stale. When an on-call agent learns a cluster moved from us-east-1 to eu-west-1, the old belief is not merely outdated, it is false, and an agent that still retrieves it will take wrong actions on production systems.

Retracting a stale belief is not a delete. You need the old belief for the audit trail, you need to know what replaced it and why, and the retraction and the replacement have to become visible at the same instant.

That last requirement is what makes this a CockroachDB project rather than a prompt. With a separate vector store there is a window between "retract the row" and "delete the vector", and during that window semantic search returns a belief the agent has already abandoned. You cannot close that window, because no transaction spans both systems.

CockroachDB closes it.

What it does

An on-call SRE agent whose memory is CockroachDB. It recalls what it already believes before investigating, reads live cluster state, records new observations, and when an observation contradicts memory the stale belief is retracted atomically with provenance and a written rationale. It also refuses to execute an irreversible remediation twice, even when killed mid-action.

How we built it

The claim, its embedding, its validity interval, its supersession pointer, its retraction reason and its provenance are columns in one CockroachDB row. A revision is therefore one transaction: insert the replacement, close the old belief's validity interval, write the revision record. There is no instant where recall can see one without the others.

Adjudication is deliberately outside that transaction. Deciding whether two claims conflict needs a model call, and holding a database transaction open across a network round trip to an LLM is how a memory layer becomes an outage. So the flow is plan, adjudicate unlocked, then commit, and the commit re-reads every conflicting belief FOR UPDATE and rejects itself if another run revised it meanwhile.

Durable execution is the second half. Intent is committed before any side effect, keyed by a hash a replay reproduces exactly, with a UNIQUE constraint enforced by the database rather than an application check a race can slip through.

CockroachDB tools used

Distributed vector indexing. beliefs.embedding is a VECTOR(384) column with a cosine vector index, and it is the memory substrate. Enabling it works on the free Basic tier.

The finding worth reporting: the first implementation put scope filters in the same query as the ordering, and CockroachDB will not serve a vector index when other predicates are attached to the ORDER BY. It planned as a full scan. Every test still passed, because results are identical and only the access path was wrong, which means the project's headline claim was false in the query plan while every check was green. Found by reading EXPLAIN, not by testing. The shipped query nests the search and plans as vector search table: beliefs@beliefs_embedding_idx, with a regression guard that asserts on the plan and includes a discrimination check proving the old shape still does not use the index.

ccloud CLI. Wraps the control plane. Arguments are passed as an argv array to execFile, never interpolated into a shell string, so there is no command injection surface even if the model emits adversarial text. --json forced. Read verbs run directly; anything mutating is refused unless it carries a durable action log intent.

Agent Skills repository. The 34 official skills are ingested as beliefs with source_kind='skill', 105 claims after heading-level chunking. Not a bibliography: they sit in the same table as lived observations, compete in the same semantic search, and are subject to the same revision rules, so a skill that does not apply to this cluster gets superseded by observation like any other stale belief.

Managed MCP server. Client implemented with a client-side read-only allowlist. The server exposes create_database, create_table and insert_rows; Palimpsest refuses to call them, rejecting anything outside the allowlist before the request leaves the process and writing the refusal to the audit log. Authority bounded by code, not by trusting a remote default. Reports not-ok on the live deployment because no service account key is configured there; the other three tools are exercised live.

AWS services used

Amazon EC2 runs the agent and its HTTP surface. Amazon S3 stores deployment artifacts and evidence bundles referenced by beliefs.

Bedrock was the original design and is unavailable on this AWS account: every model in every region returns ValidationException: Operation not allowed while new-account verification is pending, and Anthropic models return an additional geographic restriction. The rules require at least one AWS service rather than Bedrock specifically and state any combination of models may be used, so embeddings run in-process with all-MiniLM-L6-v2 and reasoning goes to an OpenAI-compatible endpoint behind a single module. Switching back is one module, one config value, and re-running the dimension migration.

A Lambda Function URL adapter over the same request handler is in the repository. It is not the live deployment because the deploying principal has PowerUserAccess, which excludes iam:CreateRole, and Lambda cannot be created without an execution role. EC2 needs no role.

Challenges we ran into

Four bugs, and the interesting thing is that passing tests actively concealed three of them.

The vector index was never used (above). Green tests, false claim.

Quarantine rolled back. An ambiguous action was marked QUARANTINED and the error thrown from inside the same transaction callback, so the update reverted. The agent still refused to re-execute, which is why exactly-once assertions passed, but the row never persisted and an operator would find no record of the action needing reconciliation.

The contradiction threshold was unfixable as a number. It started at 0.82, chosen by intuition. Calibration against labelled pairs: real conflicts span 0.51 to 0.88 while compatible pairs reach 0.81, so the classes overlap by 0.29 and no single threshold separates them. At 0.82, three of five real conflicts were missed. Reproduced live: a paraphrased contradiction was stored as a second belief, leaving two mutually exclusive claims about one cluster both retrievable, which is exactly the failure this project exists to prevent. The fix uses the column beside the vector: every conflict was same-subject, the worst false positive was cross-subject, so candidates now come from both indexes on the same rows.

Rate-limit backoff fought the wrong constraint. The model fallback chain assumed per-model limits; the free tier enforces 20 requests per minute account-wide across all models, so walking the chain could not clear it and converted a survivable pause into hard failure. Now it reads the reset from the provider's own header.

Accomplishments we're proud of

The verification, specifically the negative controls. The contradiction demo proves recall no longer returns a retracted belief, then proves the row still exists and is returned when explicitly requested, because without that the first assertion would pass just as happily against a DELETE or an empty table. The chaos demo ends by replaying an effect the way an agent without a durable log would and asserts the count does double, which is what proves the earlier assertions were not vacuous.

Suites: contradiction 13/13, chaos 8/8, agent loop 8/8, paraphrase 7/7, WAL 7/7, index plan 5/5, schema 6/6.

What we learned

A green test suite is not evidence about the thing you think it is. Three of the four real bugs were invisible to passing tests and visible only in the query plan, in database state after a crash, and in calibration data.

What's next

A larger labelled corpus for precision tuning. Exercising the serializable retry path against an induced CockroachDB failover, which the chaos demo does not cover. Per-namespace partitioning so the over-fetch tradeoff in recall stops being a tradeoff.

Built With

Share this project:

Updates