Recall

Recall is branchable, replayable memory for AI agents. Fork memory before risky work, then prove which stored fact drove any past decision. Built on CockroachDB.

Comment

Recall

Branchable, replayable agent memory on CockroachDB and AWS.

Live demo: https://0ghwemzkvk.execute-api.eu-west-1.amazonaws.com

Inspiration

Two things kept bothering me about agent memory.

The first is that agents can't speculate safely. An agent that explores a bad path writes that path into its memory permanently. There's no branch, no rollback, no way to say "try this, and if it goes badly, throw the whole thought away." Every other part of software engineering solved this decades ago with version control. Agent memory is still a flat append-only log.

The second is worse. When an agent does something wrong, nobody can answer the only question that matters: what did it believe at the time, and which stored fact caused this? Without that, you can't ship an autonomous agent into anything regulated or high stakes. You're asking people to trust a decision you can't reconstruct.

So I built the thing that answers both. Fork memory before risky reasoning. Commit or discard the branch. And for any past decision, replay the exact memories that drove it.

What it does

An agent forks its memory into a branch, reasons inside it, and records its decision along with the precise set of memories it retrieved to get there. Later, when one of those memories turns out to be wrong and gets withdrawn, the decision inspector flags it. Then you hit rewind and get two answers side by side: the same decision replayed against the exact memories the agent originally had, and the same question run against what we know now. When those two differ, you've proven the memory is what changed the outcome.

The demo scenario is incident triage. A database slows down, the agent recalls prior incidents, forks a branch, and follows a runbook. Later that runbook gets withdrawn. The faithful replay reproduces the original call. The re-run against today's memory picks something else entirely.

There's also a timeline scrubber for moving back through what a branch knew at any past instant, and when you go past what the cluster can honestly reconstruct, it tells you instead of guessing.

How we built it

Python kernel, and every line of SQL lives inside it. Nothing else in the project talks to the database, which sounds like a style preference until you realize it's what makes the audit log trustworthy. Every write produces an audit row in the same transaction as the write itself. I have an executable test that enforces this, and writing it found two genuine leaks I'd missed.

CockroachDB does the actual hard work:

  • Branch reads use logical time travel over validity columns, resolved through a recursive CTE plus the vector index in one statement
  • Forensic replay uses AS OF SYSTEM TIME
  • Concurrent agents forking and committing are safe because serializable isolation is the default, not an opt-in
  • Semantic retrieval runs on the distributed vector index, scoped to a branch

On AWS: Lambda runs the agent and the API, API Gateway fronts it, S3 holds snapshots and audit archives, CloudWatch handles logs and alarms. All of it co-located with the cluster in eu-west-1, for reasons in the challenges section.

I used four CockroachDB tools. Distributed vector indexing for retrieval. The Cloud Managed MCP Server, deliberately read only, for schema introspection during development, so it could never bypass the audited write path. The ccloud CLI for provisioning and audit export. And I published a Recall skill back in the Agent Skills format.

Claude Code wrote about 97% of the code. My job was architecture, decisions, and refusing to accept results I couldn't verify.

Challenges we ran into

Anthropic models are geo blocked from my region. I'm in Lagos. Bedrock returns a ValidationException for any anthropic.* model, and deploying to Lambda in us-east-1 didn't help, because the restriction follows the account rather than the caller. I proved it with a throwaway Lambda before building anything on the assumption. Reasoning runs on NVIDIA's Llama nemotron-49b instead, which fit under API Gateway's 30 second ceiling and picked the correct runbook 5 out of 5 times where the 8B model didn't.

Titan embeddings have an applied quota of zero, and it's not adjustable. I tested whether it was a burst limit I could pace around: six and a half minutes of cumulative spacing, every call after the first throttled. Then I read the applied value rather than the AWS default and found Value=0.0, Adjustable=False. There's no Service Quotas form to file against that. So embeddings run on a stand in provider, the health endpoint reports it live, and I don't claim semantic recall quality anywhere.

A vector space mismatch nearly shipped silently. My own run script recommended seeding offline and recording live, which puts fake vectors and real Titan vectors in the same table. Cosine between two unrelated spaces is roughly zero, so recall returned six confident looking results with plausible scores that were pure noise. Nothing errored. The only symptom was "the numbers look a bit low." Every memory now records which embedding space produced it, and four startup paths refuse a mismatch.

The console was never deployed. verify_deployment.sh passed 26 out of 26 while the demo URL returned a 404, because every check it made was an API check. "The deployment works" and "the demo works" were different claims and only one of them was being tested.

Three thousand miles of latency. The rewind endpoint timed out at the gateway and I assumed it was the model. It wasn't. The cluster sits in eu-west-1 and the Lambdas were in us-east-1, so roughly 20 sequential kernel queries per arm crossed the Atlantic. My laptop in Lagos was faster than the Lambda in Virginia, which was the tell. Co-locating in eu-west-1 cut 31 seconds of replay to about 1.

Accomplishments that we're proud of

The replay is honest, and I can prove it rather than assert it. The faithful arm replays the exact recorded provenance slice, six of six memories, and the UI shows that count. When the model rewords itself, the interface says the input was identical and the wording varied, instead of implying the replay failed.

The concurrency story is demonstrated, not claimed. Racing commits produce exactly one clean winner and one structured conflict, with a real 40001 retry in the logs. Eight concurrent writers, zero lost updates, and audit rows exactly equal to successful operations.

The vector index survives contact with reality. EXPLAIN on the real production query shows a vector search node over real rows, and the plan under AS OF SYSTEM TIME is byte identical to the live plan, so time travel is transparent to the planner's index choice.

I never faked a result to make a demo look better. When Titan wouldn't run, the health endpoint said so. When a badge would have implied more than I could prove, I changed the badge instead of the claim.

What we learned

AS OF SYSTEM TIME can't do what I designed it to do. My original plan had branch reads reconstructing ancestry with per fork point timestamps. That's structurally impossible: AOST is a top level statement modifier, so one statement carries one timestamp. You can't say "read this branch live, its parent as of fork one, and its grandparent as of fork two" in a single query. Branch reads became logical time travel using validity columns, which is pure SQL and works at any age. AOST got reserved for forensic replay, where a single whole statement timestamp is exactly right. Two mechanisms for two jobs, and the result is better than the plan was.

branch_id IN (SELECT ...) quietly kills the vector index. The planner full scans. Resolving ancestry into a literal ID list first preserves the index prefix spans. I caught it by running EXPLAIN on the real production query rather than a synthetic one, then guarded it with a regression test seeded with 3,000 rows, because index choice is cost based and would have passed on tiny test data.

A green test suite can mean nothing at all. My conftest probed the cluster with a three second connect timeout. The cloud cluster needed six. An unreachable cluster skipped rather than failed, so over 100 database backed tests reported success while executing nothing, and the suite exited zero. I'd been shipping on that signal for three phases. Now an unreachable cluster is a hard collection error, and CI asserts a minimum pass count so the suite can't silently shrink.

Hosted models aren't byte reproducible, even at temperature zero. The same decision came back one clause shorter across runs. My fidelity check was exact string comparison, so harmless wording drift read as "the replay is broken." That forced a distinction I should have drawn from the start. "Same input" is provable from the record and is Recall's claim to make. "Same wording" is a property of a hosted endpoint and isn't anyone's guarantee.

What's next for Recall

Real embeddings, once the account quota gets sorted, and then an actual measured recall quality benchmark rather than a mechanism demo. Terraform state in S3 rather than on my laptop, and a scoped deploy role instead of root.

Beyond the cleanup, the primitive itself is the interesting part. Memory you can branch is a different thing to build on than memory you can only append to. I'd like to see what an agent does with it when speculation is genuinely free, and whether "prove which memory caused this" turns out to be useful anywhere outside the incident triage case I built it for.

Built With

Share this project:

Updates

Submission history