Cairn

always-on memory for AI agents. semantic recall over a CockroachDB distributed vector index, and it survives a wipe. built on AWS Bedrock, Lambda and S3.

Comment

Cairn

An always-on memory backbone for a multi-agent operator. Built on CockroachDB, deployed on AWS.

By Ayman Dakir. Personal project. MIT licensed. No branding, no client data, just infrastructure.

Inspiration

my agent memory got wiped by a bad merge in June. one careless squash and a fleet of agents forgot who they were, what they had learned, and what they were in the middle of doing.

I run a 14-agent autonomous setup for real work. Along the way I kept a catalogue I keep of 20 concrete ways agents break in production. Stale memory. Context rot. Silent state drift. Coordination that assumes a shared brain that does not actually exist. Most of these are not model problems. They are memory problems.

So I stopped treating memory as a file next to the code. A file can be merged over, truncated, or lost. I moved agent memory somewhere that cannot happen. CockroachDB, always-on, distributed, survivable. This is the version of that I can deploy.

Not a framework. A backbone.

What it does

Cairn gives a group of agents one shared, durable memory with four jobs.

  1. Key-value state per agent. Each agent stores and reads its own working state. agent_state, keyed by (agent_id, k).
  2. A coordination and run-log bus. Agents post events and tasks to memory_log. Another agent reads the log and acts. This is how they hand work off without a private channel.
  3. Semantic recall. Every durable fact is embedded and stored in memories with a 1024-dim vector. Any agent can ask a question in plain language and get the top-k most relevant memories back, across every agent or scoped to one.
  4. Survivability. State, memories, and log snapshot to S3 as JSON. If the rows get wiped, restore rehydrates from the latest snapshot and recall keeps working. This is the June incident, turned into a button.

The demo runs three named agents: scout, builder, publisher. They each remember a few facts, coordinate through the log, recall across each other to answer a question through Bedrock, then survive a simulated wipe.

How it is built

TypeScript, Node 20, ESM. Runnable by one developer with a database URL and AWS creds. No secrets in code, everything through env.

  • src/db.ts a singleton pg Pool from DATABASE_URL, TLS set for CockroachDB Cloud. migrate() runs schema.sql.
  • src/bedrock.ts embed() calls Titan v2 for 1024-dim vectors. complete() calls the Bedrock Anthropic messages API for the reasoning step.
  • src/s3.ts putJson, getJson, latestKey against MEMORY_S3_BUCKET.
  • src/memory.ts the core module. registerAgent, setState, getState, logEvent, readLog, remember, recall, snapshot, restore.
  • src/agent.ts the CLI demo, npm run demo.
  • src/lambda.ts an API Gateway HTTP API v2 handler exposing POST /remember, POST /recall, GET /log, GET and POST /state.
  • public/index.html one self-contained file. Write a memory, run semantic recall with scores, watch the coordination log, and hit wipe and restore to prove memory survives.
  • src/memory.test.ts node:test unit tests with pg and Bedrock mocked, so they run with no live creds.

The schema is boring on purpose. agents, agent_state, memory_log, memories. The interesting column is one line: embedding VECTOR(1024).

CockroachDB tools used, and what the agents did with them

Two CockroachDB capabilities, both load-bearing.

1. Distributed Vector Indexing (C-SPANN)

memories carries a 1024-dim embedding per row. The index is one statement:

CREATE VECTOR INDEX memories_embedding_idx ON memories (embedding);

This is CockroachDB's distributed vector index, C-SPANN, which needs v25.2 or later. Recall runs cosine distance with the <=> operator:

SELECT id, agent_id, source, content,
       1 - (embedding <=> $1) AS score
FROM memories
WHERE ($2::string IS NULL OR agent_id = $2)
ORDER BY embedding <=> $1
LIMIT $3;

The embedding of the query goes in as $1. Scope is optional through $2. In the demo, publisher asks a question and the top-k rows come back from facts that scout and builder wrote. Score is 1 - cosine distance, so higher is closer. The vector search is not a bolt-on service. It lives in the same transactional database as the state and the log, so recall and coordination read from one consistent place.

2. CockroachDB Managed MCP Server

Endpoint https://cockroachlabs.cloud/mcp. I use it for schema and operational work around the database, config and docs, not called from application code. It is how I inspect the cluster, confirm the vector index exists, and check the version supports C-SPANN, from an agent-facing surface instead of a separate console. The application talks to CockroachDB through pg; the MCP server is the management plane.

Two tools, both doing real work. One holds the memory and answers vector queries. One manages the cluster the memory lives in.

AWS services used, and how

  • Bedrock. Titan Text Embeddings v2 (amazon.titan-embed-text-v2:0) produces every 1024-dim vector, on remember and on recall. Claude 3.5 Sonnet (anthropic.claude-3-5-sonnet-20241022-v2:0, configurable) does the reasoning step where publisher turns recalled memories into an answer.
  • Lambda. src/lambda.ts is an API Gateway HTTP API v2 proxy handler. It exposes remember, recall, log, and state as JSON endpoints with permissive CORS so the demo UI can call it directly. The whole backbone runs serverless.
  • S3. Snapshots of state, memories, and log land in MEMORY_S3_BUCKET as timestamped JSON. restore reads the latest key and rehydrates. This is the durability floor under the wipe-and-restore demo.

Region us-east-1 throughout.

Challenges

The honest ones.

  • Vector index version floor. C-SPANN needs a recent CockroachDB. On an older cluster the index statement is the thing that fails, so the migration has to target v25.2 or newer.
  • Embedding dimensions have to match the column. Titan v2 at 1024 and VECTOR(1024) are one decision, not two. Change the model, change the schema.
  • Making the wipe convincing. A restore that quietly re-embeds would be cheating. The demo deletes rows, shows recall failing, restores from S3, then shows the same recall working. The proof is that recall recovers, not that the code ran.
  • Testing with no creds. The unit tests mock pg and Bedrock so anyone can clone and run them without an AWS account or a live cluster.

What is next

  • Memory decay and pinning, so stale facts fade and load-bearing ones do not.
  • Per-agent recall scoping already exists through $2; next is richer policy, who can read whose memory.
  • Snapshot cadence and point-in-time restore, not just latest.
  • Working through more of that catalogue. This build closes stale memory and context loss. There are 18 other failure modes on the list.

Testing instructions for judges

Everything runs from a clone. Two paths: fully local unit tests with no creds, or the live demo with a cluster and AWS.

Environment

DATABASE_URL=postgresql://<user>:<pw>@<host>:26257/cairn?sslmode=verify-full
AWS_REGION=us-east-1
BEDROCK_EMBED_MODEL=amazon.titan-embed-text-v2:0
BEDROCK_CHAT_MODEL=anthropic.claude-3-5-sonnet-20241022-v2:0
MEMORY_S3_BUCKET=<your-bucket>

The cluster must be CockroachDB v25.2 or newer for the vector index. AWS creds come from the standard AWS SDK chain. No secrets go in code.

1. Tests with no live creds

npm install
npm test

pg and Bedrock are mocked, so this passes offline.

2. Migrate the schema

npm run migrate

Creates agents, agent_state, memory_log, memories, and the vector index memories_embedding_idx.

3. Run the demo

npm run demo

Watch, in labeled steps: scout, builder, and publisher each remember facts; one posts a task to memory_log and another reads it and acts; publisher recalls semantically across the others and answers through Bedrock; then snapshot, wipe the memory rows, restore from S3, and recall the same question to prove memory survived.

4. Live URL

Open public/index.html and set the API base input to your deployed Lambda URL (or run the handler locally). The page lets you write a memory, run recall with visible scores, watch the coordination log, and hit wipe and restore.

Mapping to the five judging criteria

  1. Technological implementation. Two CockroachDB tools (Distributed Vector Indexing / C-SPANN and the Managed MCP Server) and three AWS services (Bedrock, Lambda, S3). State, coordination, and vector recall live in one transactional database. Typed TypeScript, tested offline.
  2. Design. One core module with a small, exact API. A self-contained UI, no build, no required CDNs. The schema is four tables and one vector column.
  3. Potential impact. Multi-agent systems lose memory in production and it breaks them silently. Cairn moves memory to a place a bad merge cannot erase and proves recovery with a button. It comes straight from a real 14-agent operation and a documented catalogue of 20 failure modes.
  4. Quality of the idea. Not a chatbot with a database. A survivability layer for agent memory, motivated by an actual June wipe, closing two named failures: stale memory and context rot.
  5. Use of CockroachDB and AWS. Named queries, named operators, named models. The vector index is CREATE VECTOR INDEX memories_embedding_idx. Recall orders by embedding <=> $1. Embeddings are Titan v2 at 1024 dims. The API is a Lambda. Durability is S3. Nothing decorative.

Built personally, in the open. a cairn is the marker you stack so the next traveler finds the way.

Built With

Share this project:

Updates