RecallOps : Autonomous Incident Investigation Agent

An autonomous SRE agent with persistent memory that survives crashes, resumes investigations from where it stopped, recalls past incidents, and learns from every resolution.

Comment

Inspiration

AI agents can reason through complex tasks, but their progress is often tied to the process running them. If that process crashes, the agent may lose its context and have to start over.

We wanted to explore a simple question: Can an AI agent remember what it was doing even after a crash?

That idea led us to build RecallOps — an incident investigation agent with durable working memory.

What it does

RecallOps helps investigate operational incidents through a persistent investigation workflow.

It can:

  • Create and track incidents
  • Start an investigation and move through controlled states
  • Generate structured reasoning, hypotheses, evidence, and next actions
  • Persist the agent's working memory in CockroachDB
  • Record investigation state history
  • Recover an investigation after an application crash
  • Resume from the last persisted checkpoint instead of starting from scratch

The MVP uses a deterministic reasoning engine so the crash-and-recovery behavior is reliable and reproducible.

How we built it

We built RecallOps using Java, Spring Boot, React, TypeScript, and CockroachDB.

The backend separates incident management, investigation lifecycle, agent reasoning, persistent working memory, and recovery into independent components.

The key design decision was to treat the investigation as a durable state machine. Important reasoning context is checkpointed to CockroachDB so it survives application failure.

The frontend provides an interactive way to create incidents, run investigations, inspect memory, and demonstrate the crash/recovery workflow.

Challenges we ran into

The biggest challenge was deciding what an agent actually needs to remember in order to continue an investigation after a restart.

We had to design a reliable state model for investigations, persist working memory separately from execution, prevent invalid state transitions, and make recovery deterministic enough to demonstrate clearly.

We also had to keep the MVP focused rather than trying to build a complete production SRE platform.

Accomplishments that we're proud of

We are proud that RecallOps demonstrates its core idea end-to-end:

Create incident → investigate → persist reasoning → crash application → restart → recover memory → continue investigation.

The most important achievement is that the investigation itself survives the failure of the process running it.

We also built the persistence layer, state machine, recovery service, REST APIs, frontend, and automated tests as one working MVP.

What we learned

We learned that building reliable AI agents is not only about improving the intelligence of the model.

It is also about engineering the system around the model.

Persistent memory, explicit state transitions, checkpointing, recovery, and clear separation between ephemeral execution and durable state are essential when agents perform long-running tasks.

What's next for RecallOps: Autonomous Incident Investigation Agent

The current MVP uses deterministic reasoning to prove the persistence and recovery architecture.

Next, we want to evolve RecallOps into a truly autonomous SRE agent by adding:

  • LLM-powered reasoning
  • Tool calling for logs, metrics, and services
  • Historical incident retrieval and vector search
  • Automated root-cause analysis
  • Multi-step investigation planning
  • Automated remediation suggestions
  • Agent evaluation and observability
  • Multi-agent collaboration

The long-term goal is simple:

An autonomous SRE agent that can investigate incidents, remember its reasoning, survive failures, and continue working without losing its context.

Built With

Share this project:

Updates

Submission history