RecallOps

Governed SRE memory that recalls proven outcomes—not merely similar incidents.

Comment

Inspiration

Operational agents often treat vector similarity as permission. That is dangerous: the closest past incident may belong to another tenant, target an incompatible release, or recommend an action that previously made an outage worse. RecallOps starts from a stricter idea: memory is evidence, and evidence must earn the right to influence an action.

What it does

RecallOps is a governed incident-response agent for SRE teams. It analyzes an incident, retrieves semantically related operational memories, and applies deterministic tenant, validity, compatibility, governance, and observed-outcome controls before ranking them. Vector similarity discovers candidates; it never authorizes them.

The agent then proposes a typed action. Mutating proposals require approval of the exact action. After an operator attests the externally performed action and records the result, RecallOps creates a memory in pending_review. That memory remains retrieval-ineligible until a different Amazon Cognito identity activates it. The next incident can recall the same reviewed memory with its source, outcome, and governance provenance. Revocation removes a memory from active retrieval immediately while preserving its audit history.

RecallOps does not claim to execute infrastructure mutations in this release. It acts on memory by selecting or safely rejecting precedent, producing an evidence-bounded proposal, enforcing approval, and learning from governed outcomes.

How we built it

CockroachDB is the persistent memory system, not a replaceable metadata store. One transactional causal record connects incidents, JSON analyses, 1,024-dimensional embeddings, candidate decisions, approvals, execution attestations, outcomes, memory state, and governance events. Relational constraints enforce tenant-scoped provenance and idempotency without synchronizing a separate vector database.

We meaningfully used two qualifying CockroachDB tools:

  1. Distributed Vector Indexing generates semantic candidates inside tenant, service, and embedding-space boundaries through memories_embedding_v2. Deterministic policy then decides eligibility and rank.
  2. CockroachDB Agent Skills Repo: the pinned official designing-application-transactions skill directly shaped atomic incident-plus-outbox persistence, provider calls outside transactions, and bounded serialization retries.

The public application runs on Amazon ECS Fargate behind API Gateway, a private VPC Link, and an internal load balancer. Amazon Cognito verifies identity and separates operator and reviewer roles. Amazon S3 stores encrypted, versioned evidence; CloudWatch supplies logs, metrics, alarms, and server-verifiable alarm evidence; Secrets Manager injects database credentials.

The public judge path deliberately uses deterministic reasoning and deterministic embeddings so model-account authorization cannot break evaluation. Amazon Bedrock is supported as an optional bounded provider, but it is not active in the submitted deployment.

Challenges we ran into

The hardest problem was preventing semantic retrieval from becoming an authorization bypass. A highly similar memory may belong to another tenant, apply to another release, have failed previously, be unreviewed, or have been revoked. We aligned query boundaries, compatibility policy, four-eyes governance, retries, and audit provenance so concurrent or repeated requests converge on one consistent result.

We also made evidence claims fail closed. Production-strength claims require server-verifiable AWS evidence; local attestations remain explicitly labeled.

Accomplishments that we're proud of

  • A complete store → govern → retrieve → propose loop with one stable memory ID visible through the UI and CockroachDB.
  • Safe abstention when no reviewed, compatible successful precedent exists.
  • Immediate revocation from active recall while audit history remains.
  • Four-eyes review enforced using verified identities, not a UI convention.
  • An immutable AWS release with 100% combined Python statement and branch coverage, live CockroachDB integration tests, browser accessibility tests, concurrency tests, and a safety-critical mutation-testing gate.
  • Reproducible evidence with limitations stated instead of inflated claims.
  • A separate open-source memory-retrieval safety contribution proposed upstream; it is not counted as the accepted official Agent Skill used for eligibility.

What we learned

Agent memory is not a transcript. It is governed evidence with scope, causality, compatibility, decay, review, revocation, and provenance. CockroachDB's union of transactional constraints and vector retrieval makes that model possible without consistency gaps.

What's next

The next increment is an allowlisted executor with least-privilege task roles and automatic postcondition collection. We would also expand independent human evaluation and managed-cluster scale testing.

Built With

Share this project:

Updates