DejaOps

An on-call copilot whose entire memory is CockroachDB: vector recall of past incidents, exactly-once remediation in one transaction, and time-travel replay of what it knew.

Comment

Inspiration

Every on-call engineer knows the 3 AM feeling: checkout is down, the graphs look familiar, and somewhere in a Slack thread from eight months ago is the answer — written by an engineer who has since left the company. Institutional memory lives in people, and people churn.

So I set out to build an agent whose memory outlives the team. What I found while building it is that recall is the easy part. The hard part is trust:

  • An agent that retries a timed-out tool call will happily restart the service twice, or refund a customer twice.
  • A postmortem asks "what did the agent know when it acted?" — and a vector store can only tell you what it knows now.
  • A scratchpad that never forgets becomes a landfill that poisons every future search.

Every one of those is a transactional database problem wearing a vector-search costume. That reframing became the whole project: don't bolt a vector store onto your agent — put the vectors inside the transactional database that already holds its state.

What it does

DejaOps is an on-call incident copilot that triages alerts against everything it has ever seen.

When an alert fires, it searches episodic memory (every past incident) and semantic memory (distilled runbooks) before proposing anything. In the live demo, an alert phrased as "customers cannot complete purchases, /charge taking over 9 seconds, began shortly after this afternoon's release" — sharing almost no vocabulary with anything in memory — recalls a three-month-old incident at 71.5% similarity and answers unprompted: "We have seen this exact pattern before. Root cause: a deploy introduced an N+1 query in fraud-check that exhausted the connection pool. Resolution: rollback fixed it in 2 minutes. Confidence: HIGH."

From there:

  • It records findings in working memory that expires on its own via CockroachDB row-level TTL — hypotheses decay, knowledge persists.
  • It remediates through an exactly-once action ledger. Tell it "the rollback timed out, run it again" and it returns DUPLICATE SUPPRESSED — the side effect was NOT re-executed, because the action and its memory record committed in a single CockroachDB transaction keyed by a unique idempotency key.
  • On resolution it consolidates: the incident becomes an episodic memory chunk, and Claude distills a reusable runbook into the semantic tier — so the next engineer inherits the lesson, not just the log.
  • For the postmortem it time-travels. AS OF SYSTEM TIME replays exactly what the agent knew at any moment during the incident.

The UI deliberately exposes the memory layer rather than hiding it: every recall with its similarity score, every TTL'd write, every ledger outcome. You can watch the database do the work.

How we built it

CockroachDB is the entire memory layer — six tables, no second datastore:

Table Role
memory_chunks VECTOR(1024) + CREATE VECTOR INDEX (tier, embedding) — episodic and semantic recall
working_memory Row-level TTL (4h) — the decaying scratchpad
action_ledger UNIQUE idempotency_key, written in the same transaction as the memory record
memory_versions Append-only, written on every mutation — backs replay beyond the GC window
incidents / incident_events / runbooks Relational agent state, transactionally adjacent to the vectors

Inference and embeddings: Claude Haiku 4.5 drives a hand-written tool-use loop (five memory tools: recall episodic, search runbooks, record finding, execute remediation, resolve incident). Every tool call is traced to the UI. Embeddings are 1024-dim and L2-normalized. Both are routed through a provider switch — Amazon Bedrock (Claude + Titan Text Embeddings V2) or the vendors' own APIs — selected by two environment variables.

AWS: an AWS Lambda container image behind an API Gateway HTTP API, secrets in SSM Parameter Store, image in Amazon ECR, structured logs in CloudWatch. No VPC, no NAT Gateway, no always-on compute — idle cost is effectively zero.

Engineering details that mattered: psycopg3 with a retry loop on SQLSTATE 40001, because CockroachDB is SERIALIZABLE and surfaces contention to the client — and those retries are precisely the case the idempotency ledger exists to absorb. Offline modes (deterministic hash embeddings + a canned LLM) keep the whole stack and its test suite runnable with no vendor credentials, which turned out to be the difference between shipping and not.

Disclosure, per the rules: AI coding assistants (Claude Code) were used throughout development.

Challenges we ran into

Only L2 distance is index-accelerated in CockroachDB. Cosine isn't. Rather than give up the index, I L2-normalize every embedding to unit length — on unit vectors, L2 ranking is monotonically equivalent to cosine ranking, so the accelerated path yields cosine semantics for free.

AS OF SYSTEM TIME can't reach past the garbage-collection window. A time-travel demo built naively works on the day you build it and silently fails on the day you record. So replay is backed by an append-only memory_versions table written in the same transaction as every memory mutation, and the API reports which mechanism answered.

Vector indexes must exist before the data does. Creating one on a populated table blocks writes, and IMPORT INTO isn't supported on vector-indexed tables — so the schema is created empty and seeded with plain INSERTs.

A brand-new AWS account is a minefield. Bedrock model access required an allowlisting review that hadn't cleared by submission time. Lambda Function URLs returned 403 regardless of a correct AuthType: NONE and resource policy (API Gateway sidestepped it). Docker 29 pushes OCI manifests with attestations that Lambda rejects. libpq's verify-full ignores the system trust store, so the CockroachDB CA had to be baked into the image — a connection that worked locally failed in Lambda with a bare "certificate verify failed". And an expired ECR token made docker push 403 on a blob, after which update-function-code silently redeployed the old image — so code changes appeared to do nothing.

The bug I'm most glad I caught. During rehearsal the agent's déjà-vu explanation kept vanishing. The loop was overwriting the reply text on each iteration, and the narration that explained what memory returned arrived in an early iteration alongside the tool calls. The single most important sentence in the product was being thrown away every turn.

Rehearsing a memory system pollutes it. Each take resolved an incident, which consolidated a new chunk and a generated runbook that then competed with the curated corpus in recall. That's the product working correctly — so it needed a reset script rather than a fix.

Accomplishments that we're proud of

The exactly-once ledger. "An agent that retries a tool call must not perform the side effect twice" is a real, unsolved pain in production agents — and it falls out almost for free when your memory layer is a serializable SQL database instead of a bolt-on vector store. It's verified by integration tests against a live cluster: after a retry, exactly one ledger row, one side effect, and one memory write.

Time-travel replay of an agent's knowledge state. I haven't seen another agent that can answer "what did you believe at 02:14, and why did you act that way?" It's a genuinely new postmortem primitive, and it exists because the memory layer is a database with MVCC.

Honest engineering under pressure. When Bedrock access didn't clear, the answer wasn't to fake it or quietly claim it — it was a provider switch that keeps the (verified) Bedrock path first-class, plus a README that says exactly what ran during the recording and how to flip back.

What we learned

Agent memory is a transactional-database problem in disguise. Vectors next to your system of record mean sync pipelines, indexing lag, and two sources of truth. Vectors inside it mean a memory is searchable the instant its transaction commits, and every claim the agent makes is auditable in time.

Concretely: SERIALIZABLE isolation and retry-on-40001 aren't obstacles to building an agent — they're the mechanism that makes exactly-once tool execution possible. And row-level TTL turned out to be a cognitive feature, not a storage optimization: forgetting is what keeps recall sharp.

The operational lesson was blunter. Every hour lost went to infrastructure whose failure mode lied to me — a 403 that meant "new account restriction", a certificate error that looked like an allowlist problem, a deploy that appeared to succeed while running stale code. Verify the artifact you deployed before you debug the code you wrote.

What's next for DejaOps

  • REGIONAL BY ROW multi-region memory — pin an EU customer's incident memories to EU nodes for data residency. The schema is one ALTER TABLE away; I skipped it because a single-region deployment I could actually demo beat a topology I could only claim.
  • Changefeed-driven consolidation — move runbook distillation off the request path into a background worker fed by a CockroachDB changefeed.
  • Real integrations — PagerDuty and Datadog alert ingestion, and remediation executors that call Kubernetes and the deploy system instead of simulating them. The exactly-once mechanics around them don't change, which is the point.
  • Memory quality feedback — let engineers mark a recalled incident as relevant or not, and use that signal to re-rank future recall.
  • Cross-team memory with row-level access control — one memory layer, many teams, each seeing only its own services' history.

Built With

Share this project:

Updates

Submission history