Amnesia-Proof Agent

Inspiration

Agent memory is usually a vector store bolted onto the side of an app. That works until the availability zone holding it goes away - and then the agent doesn't degrade gracefully, it stops. It forgets the task it was mid-way through, or worse, re-runs a side effect it already performed because the record of having done it is gone.

That failure mode is a database problem wearing an AI costume, and CockroachDB already solved it for operational data. So we stopped treating "agent memory" as a special kind of storage and put every tier of it inside a database that survives regional failure by design.

What it does

Amnesia-Proof Agent is an on-call database reliability agent whose entire memory - working state, semantic recall, task checkpoints, and the audit trail - lives in CockroachDB.

It learns. Given differently-worded reports of the same incident class, it initially falls back on an innate playbook ("page a human"). After three close episodes it consolidates them into a durable lesson, and from then on acts on its own experience instead of the default.

Then we kill an entire region on camera. The agent keeps working: it handles the next incident, writes new memories, and recalls lessons that are homed in the region that just died - served from surviving replicas. The hash-chained audit log comes through the outage with zero gaps.

Memory tiers mapped to CockroachDB primitives

Tier	Primitive	Why a bolt-on vector store can't do this
Working state	serializable transactions	state + embedding + audit row commit atomically, or not at all
Semantic recall	VECTOR INDEX (agent_id, embedding) - C-SPANN	per-agent scoped ANN, proven present in the query plan
Forgetting	Row-Level TTL (ttl_expiration_expression)	decay as a batched, admission-controlled DB job, not a cron sweep
Belief replay	AS OF SYSTEM TIME	"what did the agent believe when it acted?" - bounded by gc.ttlseconds
Survivability	REGIONAL BY ROW + SURVIVE REGION FAILURE	the receipt below
Isolation	Row-Level Security	tenant separation enforced by the DB, not a WHERE clause someone forgets
Integrity	hash-chained HMAC receipts, same txn	zero gaps across an outage, verifiable in one command

How we built it

Go, using lib/pq against CockroachDB's Postgres wire protocol. No ORM, and no AWS SDK - the S3 export is hand-rolled SigV4 (~60 lines) so the whole AWS surface fits on one screen for a reviewer. The dashboard is server-rendered HTML plus an SSE stream: no build step, no framework.

The agent loop is deliberately a deterministic rule engine plus vector recall rather than an LLM. The interesting claim is about the memory substrate, so the decision function stays reproducible - a judge running chaos.sh twice sees identical distances - and the critical path stays free of an external model API that could be rate-limited or unapproved.

Challenges we ran into

REGIONAL BY ROW silently makes crdb_region the FIRST prefix column of the vector index. So the natural reading of "constrain every prefix column to an equality" - also pin the region - is exactly wrong and costs you the index:

WHERE agent_id = $1 -> vector search (index used) WHERE agent_id = $1 AND crdb_region = $2 -> scan (index NOT used)

A kind = 'semantic' filter defeats it the same way, and a subquery-built vector defeats it where an identical literal doesn't. All three degrade silently - no error, just a full scan.

The dashboard reported a dead node as LIVE. crdb_internal.kv_node_status_and_liveness does not exist on v26.2, and membership stays 'active' for a node that is already gone - it tracks decommissioning, not reachability. Only the lease expiration moving into the past reveals a dead node.

Distance thresholds tuned against unrealistic test data. The first version fired one identical incident string repeatedly, so every same-class distance was 0.0000 and a 0.35 threshold worked. The moment inputs were made realistically varied, nothing matched and the agent silently stopped learning. Fixed by measuring the real spread (same-class 0.86-1.06, different-class 1.27) and setting the boundary from data.

Accomplishments that we're proud of

The killer receipt is reproducible by anyone in two commands:

=== KILL REGION eu-west-1 === 3 roach3:26257 is_available=false is_live=false === agent handles an incident with the region dead === {"action":"verify quorum, then rejoin the node","from_memory":true, "rationale":"recalled (dist=0.5737, home=eu-west-1)"} === chain integrity during the outage === chain VERIFIED: 50 links, 0 gaps, 0 bad hashes, 0 broken links === the revived node serves back what it never saw === WROTE DURING REGION OUTAGE

That last read executes inside the container that was killed. It wasn't running when the row was written, so serving it back proves Raft replicated and self-healed.

What we learned

Multi-region isn't a config flag you turn on - it changes your index shape, your query planner behaviour, and where your rows physically live, and it does all of that silently. Nearly every bug in this project was the database being correct while our assumptions weren't.

What's next for Amnesia-Proof Agent

Replace the local embedder with Bedrock Titan (kept off the critical path deliberately - model access approval can take days). Add CHANGEFEED-driven memory replication to an external observability sink. Contribute the consolidated agent skills upstream to cockroachdb/copilot-plugin, which currently has four empty domains and nothing on vectors or agentic memory.

Two clusters, and why

The resilience demo runs on local Docker, not CockroachDB Cloud, and this is a disclosed choice rather than a shortcut: you cannot kill a node in CockroachDB Cloud Basic/Standard - there is no DB Console, and crdb_internal is restricted by default (SQLSTATE 42501). Proving survival requires killing a node, so the chaos cluster has to be one we control. The Cloud cluster is used for the managed MCP Server and the multi-region topology.

CockroachDB tools used - 4 of 4 (two required)

1. Distributed Vector Indexing - sql/schema.sql, verified in the query plan via agent explain
2. Managed MCP Server - internal/mcp/, audited read path against cockroachlabs.cloud/mcp
3. ccloud CLI - scripts/ccloud-provision.sh
4. Agent Skills Repo - agent skills emits SKILL.md into resilience-and-disaster-recovery

Note on the MCP design: the managed server deny-lists system tables and never supports DROP/TRUNCATE, so the agent cannot route its memory layer through MCP. The correct split is MCP for the audited read/introspection path and direct pgx for transactional writes - documented in the repo.

AWS services used

S3 (audit-chain export, hand-rolled SigV4, no SDK) and Lambda (lambda/main.go, provided.al2023 custom runtime, cross-compiles to arm64).

Built With

• amazon-web-services
• aws-lambda
• cockroachdb
• docker
• go
• html
• mcp
• sse