RecallOps

Inspiration

Incident automation becomes dangerous when the automation itself fails. Stateless retries can repeat diagnosis, lose provenance, or execute the same remediation twice. RecallOps asks a narrower question: can an agent remember exactly where it stopped, preserve the evidence behind its decision, and resume without duplicating the approved action?

What it does

RecallOps is a resumable incident-memory agent for synthetic operational incidents. It retrieves scoped memory with provenance, pauses for explicit human approval, deliberately injects one execution failure, resumes the same durable run, and proves that only one action receipt is committed. The action is an allowlisted simulator. RecallOps does not connect to production infrastructure or perform real remediation.

How we built it

• CockroachDB persistent memory: incident state, source/observation provenance, approval state, idempotency keys, and durable action receipts.
• CockroachDB capability 1 — Distributed Vector Indexing (C-SPANN): relevant-memory retrieval scoped to the current synthetic incident.
• CockroachDB capability 2 — Managed MCP: an independently auditable, read-only view of durable state.
• AWS Lambda: a credential-free arm64 container image provides the bounded demo entry point.
• LangGraph: explicit checkpoints model start, approval, failure, resume, and completion.
• OpenTelemetry: structured traces expose the run and receipt lifecycle without logging secrets.

Failure-resilient flow

1. Create a synthetic checkout incident with a stable run ID.
2. Retrieve provenance-bearing memories from CockroachDB.
3. Require human approval for the single allowlisted simulator action.
4. Execute with one injected recoverable failure.
5. Resume the same run from its checkpoint.
6. Retry and verify the same receipt with execution_count = 1.

Challenges we ran into

The hard part was not calling a model; it was defining the invariants around memory and action. We had to keep retrieval tenant-scoped, preserve source and observation time, make approval durable, and enforce idempotency at the storage boundary rather than in UI state. We also separated the no-secret public demo profile from authenticated CockroachDB and Managed MCP provider verification.

Accomplishments

• A real CockroachDB provider path and a separate Managed MCP audit path were verified.
• The AWS Lambda image was deployed without credentials or production data.
• The complete synthetic flow was verified with statuses 201, 200, 503, and 200.
• The final run is complete with one committed receipt and execution_count = 1.
• The repository includes adversarial tests, trust-boundary documentation, and reproducible evidence logs.

What we learned

Agentic memory is not just vector search. A safe memory system must bind semantic recall to scope, provenance, approval, and an idempotent action receipt. CockroachDB is useful because these states can live together under transactional guarantees while still supporting vector retrieval and external read-only inspection.

What's next for RecallOps

Broaden the allowlisted simulator scenarios, evaluate multi-region failure behavior, and run an independent security review before considering any production integration.

Safety boundary

All incidents and actions are synthetic. No production systems, real remediation, real incident data, or long-lived cloud credentials are used.

Built With

• aws-lambda