AgentRecall

Time-travel replay and repair for AI agent memory — reconstruct the exact memory an agent read behind any decision, prove the cause, and fix it. On CockroachDB.

Comment

Inspiration

When a production AI agent does something wrong, you can pull up the prompt it was sent. What you can't see is the thing that actually drove the decision: what was in its memory at that moment, and why that memory won retrieval. By the time anyone investigates, the memory store has already moved on — the evidence is gone.

We kept hitting this in agent systems that read from a growing memory: the same input, run a week apart, behaves differently, and nobody can reconstruct why. We wanted to make that lost state recoverable — to point at one decision and rebuild the exact memory the agent read, prove what caused the outcome, and fix it under human authority.

What it does

AgentRecall is a forensic console for AI agent memory. Point it at a single agent decision and it:

  • Replays the exact memory snapshot the agent read — re-running the same semantic (vector) search against the past.
  • Diffs that snapshot against memory as it stands now: what was added, removed, or edited since.
  • Attributes the outcome to the memory most likely responsible, with reasons and a confidence ceiling — always a labelled hypothesis, never an automated verdict.
  • Repairs it: a human quarantines the culprit memory behind a confirmation gate, writing an append-only audit row.
  • Verifies the fix: the identical input is re-run against today's memory, and you watch the outcome change.

The demo tells a real story. An agent granted a $500 migration credit to a Starter-plan customer. At decision time, memory held only the 2024 policy (every plan eligible); the 2026 Enterprise-only policy was imported afterwards. AgentRecall reconstructs that exact past, ranks the stale policy as the cause (confidence 0.92 — higher than the corrected policy, because it was bulk-imported from a wiki), and after a human quarantines it, the same input now returns decline_credit.

How we built it

The memory layer is CockroachDB — and that is the whole mechanism, not a storage choice.

  • Distributed vector indexing (C-SPANN). Memories are embedded and retrieved by approximate-nearest-neighbour search over a real, distributed vector index. The index prefix (tenant_id, status, embedding) matches the equality filters every retrieval applies, so the optimizer chooses the index for both live and historical reads.
  • AS OF SYSTEM TIME over MVCC — the load-bearing idea. CockroachDB keeps every prior row version readable, and the vector index rides on the same MVCC storage, so a past nearest-neighbour search is replayable in one clause of SQL. This was our one unproven assumption, so it is the first thing the repo tests: top-k AS OF SYSTEM TIME T1 reproduces the recorded retrieval exactly (distance delta 0.0) while the live query diverges after mutation.
  • Serializable transactions. The memory write, the record of the action taken, and the decision itself commit in one SERIALIZABLE transaction — a crash mid-commit leaves none of them. 40001 retries are handled as expected traffic with jittered backoff.
  • Amazon Bedrock is the agent's brain. Titan Text Embeddings V2 produces the 1024-dim memory vectors and Claude is the decider (both selectable by environment variable, with a provider-provenance guard that refuses to compare vectors from different models).

Backend: FastAPI + async psycopg. Frontend: a hand-authored React/TypeScript console (no component framework). CockroachDB Cloud (aws-ap-southeast-3 / Jakarta) as the memory layer; backend on Railway, frontend on Vercel.

Challenges we ran into

  • Proving the core assumption instead of hoping. The docs don't state whether AS OF SYSTEM TIME reads work through the vector index rather than just relational rows. So we didn't assume — we wrote a gate that inserts rows, records a timestamp, mutates the index, and replays the same ANN query at that past timestamp. It passes on a live cluster.
  • The index-prefix trap. With (tenant_id, embedding) the optimizer silently fell back to a brute-force scan behind a post-filter. Adding status to match the query's equality filter fixed it — caught by reading EXPLAIN, not by trusting that the query looked right.
  • Confidence without lineage. The bad memory carried higher confidence than the correct one. That shaped a core principle: attribution is a hypothesis with a confidence ceiling, and a human — never the system — makes the repair.
  • AWS onboarding from our region. Activating an AWS account for Bedrock hit payment/identity verification snags — a real-world blocker that pushed us to make the providers pluggable and decouple hosting from AWS.

Accomplishments that we're proud of

  • A live, end-to-end forensic loop on real infrastructure — reconstruct, attribute, quarantine, verify — against a real CockroachDB Cloud cluster, not a mock.
  • The AS OF SYSTEM TIME × vector-index gate passes on a live cluster (delta 0.0), and the production replay confirms the C-SPANN index is used for both live and historical reads.
  • Zero torn states under fault injection, thanks to serializable atomicity.
  • Production-minded from day one: provider-provenance guard, human-confirmation gate, append-only audit trail, /readyz embedding-signature check, Prometheus metrics, and 41 tests.

What we learned

The sharpest lesson: confidence without lineage is the failure mode. A memory can be wrong and highly confident simply because of how it was ingested. A trustworthy agent memory has to make its own history auditable — and CockroachDB's MVCC plus AS OF SYSTEM TIME over a distributed vector index turned out to be a remarkably direct way to get exactly that: in plain SQL, with no second system to keep in sync.

What's next for AgentRecall

  • Contradiction detection — flag two active, conflicting policies before a bad decision, instead of after.
  • Regional data residency — partition the vector index with REGIONAL BY ROW (the prefix column is already in place) for multi-region memory.
  • Deeper attribution — counterfactual retrieval and causal ranking beyond the current heuristic.
  • Finish the AWS-native path — complete Bedrock activation and an App Runner / ECS deployment alongside the current setup.

Built With

Share this project:

Updates