AXIOM - crash-safe agentic memory on CockroachDB

An agent refunds $300, crashes before recording it, restarts, and refunds again. AXIOM makes that impossible: execution state and vector memory commit in one CockroachDB transaction.

Comment

Inspiration

An agent that only writes text can be wrong. An agent that moves money can be wrong expensively, and the failure is boring: the process dies between doing a thing and recording that it did it.

Almost every agent framework remembers by keeping a transcript of its own reasoning. That is fine for a chatbot and dangerous for anything that touches the world, because a transcript records what the agent was thinking, not what actually happened. Restart it after a crash and it re-reads its notes, finds an unfinished task, and does the thing again. The customer is refunded twice. Nobody notices until the money is gone.

We wanted to know what agent memory would look like if you designed it for that failure instead of around it.

What it does

AXIOM is an execution and memory layer for agents that take irreversible actions. It splits memory into four classes with different authority:

Class Answers Authority
Episodic What happened last time an agent stood exactly here? advises
Semantic What is generally true of this kind of exception? advises
Procedural Which pinned policy permits this act — or stops it? authorizes
Execution What has this agent already done, irreversibly? constrains

The first two advise. The last two can stop the agent. Vector recall tells an agent what it could do; transactional execution state decides what it may do.

The mechanism is one sentence: before the agent touches the outside world, AXIOM commits a receipt saying what it is about to do and the idempotency key it will do it under. That key is a GENERATED STORED column derived from immutable inputs, so there is no code path that can mint a different one at call time. After a crash, the recovering worker reads that receipt and re-sends under the same key, and the provider returns the original effect instead of making a second one.

The interesting part is the recovery decision itself. One serializable transaction re-checks the fencing token, point-reads the durable receipt, runs an ANN search over episodic memory for what happened the last time an agent died at this exact execution state, decides resend / escalate / re-plan, and commits the transition with its evidence attached. One commit. Memory and outcome cannot disagree, because there is no interval in which one exists and the other does not.

How we built it

CockroachDB is not the store behind the agent; it is the agent's memory, and the two jobs that matter happen in the same transaction:

  • Distributed vector indexing — VECTOR(1024) embeddings under two C-SPANN indexes, built vector_cosine_ops explicitly. Omitting the opclass silently gives L2, and a <=> query then full-scans — a wrong answer that looks like a right one. Index use is asserted from EXPLAIN at request time rather than assumed.
  • Serializable isolation with 40001 retry and full jitter, so two workers cannot both recover the same task.
  • Partial and prefix indexes — the claim index only contains outstanding work, so claim cost tracks work-in-flight rather than work-ever-done.
  • A computed prefix column for memory quarantine, so removing a memory from the index takes effect atomically at COMMIT rather than as a post-filter someone can forget.
  • Cloud managed MCP server — a model can ask the cluster questions in English, contained three ways: a SELECT-only role, a single-statement guard, and default_transaction_read_only.
  • ccloud CLI provisions the cluster and applies all five migrations.

On AWS: Lambda runs the agent workers and the API, API Gateway is the public front door, EventBridge Scheduler sweeps the queue every five minutes so the demo survives four unattended weeks of judging, CloudWatch and SNS alarm and page, and X-Ray traces the recovery path — axiom.PREPARE, axiom.dispatch and axiom.SETTLE, annotated with crash_window=W4, so the exact instant this project is about is filterable in the console.

Challenges we ran into

The provider is outside the transaction, and no amount of database rigour changes that. Exactly-once execution of a remote side effect is not available to anyone. So the guarantee is stated as what it is — effectively-once — and every crash window W1 through W7 has a documented, tested outcome instead of a hope.

Proving the memory was load-bearing rather than decorative. It is easy to build a system that recalls things and never acts differently for having recalled them. So the demo runs one stopped refund through recovery three times with the same receipt, same fence, same policy, same amount, changing only the memory table. Resend → escalate → resend. If the verdict were identical all three times, the memory here would be decoration.

Amazon Bedrock turned out to be unusable, and saying so was better than hiding it. Titan V2 answers on this account, but its on-demand quota is 0.0 requests/min, quota L-26C560CE, Adjustable=FALSE in three regions — a sustained probe got 0 of 10 calls. Batch inference needs ≥100 records and the real corpus is 10 seed texts, so padding a job to clear the minimum would have bought a checkbox by embedding meaningless strings. Embeddings run on a deterministic local model instead, and every row records which vector space it belongs to.

A table that lied about itself. axiom_memory.embedding_model had a DEFAULT naming Titan and no insert path ever set it, so every row claimed a provenance nothing had checked. Nothing computed a wrong answer — but a project arguing that systems should tell the truth about what they did does not get to ship that. Rows were reclassified by measurement, the default was dropped so forgetting the column is now an error, and a preflight gate asserts the corpus holds one vector space.

What we learned

Memory design is authority design. The useful question is not "what can the agent recall" but "what is allowed to change its mind, and what is allowed to stop it". Recall and permission are different substances, and a transcript is the worst available place to keep the record of an irreversible act.

Money is often the wrong risk axis. Forty thousand marketing emails cost about four dollars and sail through a $200 unattended ceiling; a $300 refund to one person stops and waits for a human. The cheap act is the dangerous one. So authority is denominated in the act's own unit — comms.recipients, not cents — and the second workload in the demo proves the same guarantee holds where there is no money at all.

A negative result is worth measuring properly. Bedrock's quota, the ten-times SES pricing error we found in our own source, the tracing mode we had described wrongly — each was caught by checking rather than asserting, and each is in the repo with the command that found it.

What's next

Batch-embedding through Bedrock if the account's quota is ever raised; a second real provider behind the same contract; and the Agent Skills contribution (cockroachlabs/cockroachdb-skills#23) merged rather than merely open. Open source this as well.

Built With

Share this project:

Updates

Submission history