RecallOps

RecallOps is an incident response agent that remembers what worked, learns from verified outcomes, and recommends safer fixes using CockroachDB memory and Amazon Bedrock.

Comment

RecallOps

Inspiration

The same production incident should not have to be solved from scratch twice.

An engineer gets paged at 3am because a service is down. Months earlier, someone else in the company may have seen the same failure. They already learned that restarting the pod only helped for a few minutes, while a rollback actually fixed the problem. But that experience is often buried in a closed ticket, a chat thread, or someone's memory.

So the next on-call engineer starts over. The organization already paid for that lesson once, and now it is paying for it again.

Most AI agents have the same problem. They can read an alert, search a runbook, and suggest an action, but they usually do not carry operational experience forward from one incident to the next.

That is what led me to build RecallOps.

The idea is simple:

An incident response agent should not just know what to do. It should remember what actually worked.

What RecallOps does

When an alert comes in (through Slack, teams or PagerDuty etc..), RecallOps does not start from a blank page. It first asks a practical question: have we seen something like this before, and what happened when we tried to fix it?

It searches its memory in CockroachDB for similar past incidents. What comes back is not just documentation. It is operational history: which actions were tried, which ones failed, which ones worked, and how often they worked for similar incidents.

RecallOps uses that history to score the available fixes. Restarting a service may look reasonable, but if restart has failed most times it was tried while rollback has consistently worked, that history should influence the decision.

The language model helps classify the alert and explain the recommendation in plain English. It does not decide the confidence score. The action ranking comes from recorded outcomes in CockroachDB.

RecallOps also checks what is happening right now through CockroachDB's Managed MCP Server. That gives it a live, read-only view of the environment, so the recommendation is based on both past experience and current conditions.

Once RecallOps has a recommendation, it stops and waits for human approval. Nothing changes in the environment until a person approves the action.

After approval, RecallOps runs the matching executor, verifies that the service recovered, and writes the verified outcome back into CockroachDB. That new result becomes part of the history used by future incidents.

If the evidence is too weak to support a safe recommendation, RecallOps does not guess. It escalates and leaves production untouched.

How I built it

RecallOps is built as a LangGraph workflow. One shared state object moves through the full incident lifecycle:

Triage -> Memory Search -> Reason -> Recommend -> Human Approval -> Execute -> Writeback

CockroachDB is the memory layer behind the workflow.

Distributed Vector Indexing

Past incidents are stored in CockroachDB with embeddings.

When a new alert arrives, RecallOps uses distributed vector indexing to find incidents that are semantically similar. This gives the agent long-term recall without requiring a separate vector database.

Because the vectors and the operational data live in the same system, there is also no separate store to keep in sync.

Fix History

RecallOps keeps success and failure counts for each action and incident type.

The Reason step reads that history to score the available fixes. After an approved action is executed and verified, Writeback updates the same history.

That creates the learning cycle:

Reason reads past outcomes -> action is executed -> result is verified -> Writeback updates history -> future decisions use the new evidence

Durable Checkpoints

Human approval can arrive later and through a separate request, so the workflow cannot depend on one process staying alive.

When RecallOps pauses for approval, the full run state is checkpointed in CockroachDB. When approval arrives, the agent restores that state and continues from the same point.

I tested this by stopping the agent process while a run was waiting for approval. After the process came back, the run still resumed because the state was stored in CockroachDB instead of process memory.

Managed MCP Server

Historical memory is useful, but an incident response agent also needs to understand what is happening now.

RecallOps uses CockroachDB's Managed MCP Server to read live cluster information in a safe, read-only way.

That means the agent can reason with two kinds of evidence at the same time:

what worked before and what is true right now

Amazon Bedrock

Amazon Bedrock powers the two steps that need a language model: Triage and Recommend.

Triage helps classify the incoming alert. Recommend turns the selected action and supporting evidence into a clear explanation for the operator.

Memory search, fix scoring, confidence calculation, execution, writeback, and escalation are deterministic. This keeps the operational decision easier to audit and keeps model usage focused.

The full stack runs on Amazon EC2 with Docker Compose.

Execution

Remediation is handled through an executor registry, so new actions can be added as modular executors.

For the demo, I implemented one executor all the way through: rollback.

That executor is connected to a live sandbox service. The demo shows the full path from incident detection to recommendation, human approval, execution, verification, service recovery, and memory update.

I chose to prove one real execution path instead of simulating several fake ones.

Challenges I faced

Making the pause survive a restart

The workflow has to stop and wait for human approval, sometimes for several minutes and through a completely separate request.

My first version kept the active run in process memory. That worked until the process restarted, at which point the in-flight incident disappeared.

For an incident response system, that is exactly the kind of failure I wanted to avoid.

Moving the full checkpoint into CockroachDB solved the problem and gave me a clear design rule: important run state should not depend on a single process staying alive.

Keeping the model out of the decision

I wanted the language model to help understand and explain the incident, but I did not want it inventing the operational confidence score.

It is easy for an LLM to produce a convincing number even when that number is not grounded in anything.

So I separated the responsibilities. The model classifies and explains. Deterministic code scores the candidate fixes using recorded outcomes from CockroachDB.

That boundary makes the recommendation easier to inspect and defend.

Making the explanation trustworthy

Early versions of the recommendation were technically correct but not very useful to an operator.

The explanation exposed too many raw internal numbers, and when two actions had similar scores, the model sometimes tried too hard to make one sound clearly better.

I changed the explanation so it reflects the evidence more honestly. If two actions are equally strong, RecallOps can say that instead of pretending there is a meaningful difference.

For an incident response tool, a recommendation that sounds confident for the wrong reason is worse than an honest escalation.

What I learned

The biggest lesson from building RecallOps is that agent memory becomes much more useful when it stores outcomes, not just information.

Finding a similar incident is useful. Knowing that one action failed three times while another succeeded seven times is what actually changes the next decision.

I also learned how important durable state becomes once an agent works with humans. Any workflow that waits for approval needs to survive process restarts, separate requests, and delays between steps.

Finally, I learned that keeping language generation separate from operational control makes the system easier to trust. The model can help interpret and communicate, while stored evidence, deterministic scoring, and human approval decide what actually happens.

What's next for RecallOps

Today, the demo is triggered from the RecallOps UI.

In production, the same API can connect to the tools teams already use. PagerDuty could trigger RecallOps automatically when an incident fires, or an engineer could send an incident to the agent from Slack or Microsoft Teams.

The entry point can change, but the workflow stays the same.

RecallOps is not meant to replace an organization's existing runbooks, deployment systems, or automation tools. It sits above them as a memory and decision layer that learns which runbook to trust, based on what actually happened before.

Because an incident response agent should not just know what to do. It should remember what actually worked.

Built With

Share this project:

Updates