SRE-Agent V2: Trust-Aware Agentic Memory

Incident-response AI that checks evidence before reusing historical memory and shows why records are accepted or withheld.

Comment

SRE-Agent V2

Inspiration

SRE-Agent V2 grew out of my CockroachDB-based incident-response project, which combined retrieval, analysis, and decision agents with persistent memory.

The next question concerned the memory itself: when should an agent reuse a previous recommendation?

A historical fix may belong to an obsolete configuration. A stored recommendation may also lack evidence that it ever worked. I built V2 to examine whether checking that evidence before reuse could prevent historical memory from misleading an agent.

What it does

SRE-Agent V2 validates retrieved incident and prior-agent records before passing them into analysis.

The validation layer checks recorded execution and outcome evidence, its age, service scope, and configuration compatibility. Incident fingerprints bind verification evidence to the historical content being reused. The system reduces accepted prior-agent records to ledger-backed action information.

The interface exposes acceptance and withholding decisions with their reasons.

Two views support inspection. Evidence replay compares ordinary and validated memory using saved synthetic experiments. Live proposal retrieves and validates memory, analyzes an incident, and records a proposed response.

The live workflow does not execute remediation. Recording a proposal does not establish recovery.

How I built it

NVIDIA Nemotron runs through Nebius Token Factory. AWS Bedrock Titan generates 1,024-dimensional embeddings.

AWS RDS PostgreSQL with pgvector stores incidents, embeddings, agent memory, and decisions. Separate execution and outcome-verification tables hold supporting evidence.

The hosted application uses a restricted database account that cannot write execution or verification records. AWS Secrets Manager supplies credentials.

Streamlit provides the interface. The application runs on ECS Fargate behind an Application Load Balancer and CloudFront. Viewer traffic uses HTTPS. The CloudFront-to-load-balancer connection uses HTTP protected by network restrictions and a secret origin header.

Evaluation and observed results

The controlled comparison contains 21 model calls using one synthetic payment-service incident.

Historical stale memory recommends rolling back release v2 to v0. Under the current synthetic configuration, v1 is the valid recovery target.

Across three repeats, ordinary memory passed 0/3 stale-memory resolution checks. Validated memory passed 3/3. The no-memory control also passed 3/3.

Clean-memory and poisoned-memory conditions passed all three resolution checks in both modes. No recommendations matched the declared unsafe-action criteria in these 21 calls.

These results show behavior within one controlled scenario. They do not establish reliability across incidents or show that memory validation always improves performance.

A separate hosted test completed all five workflow stages in 9.85 seconds, withheld twelve unsupported or simulation-only memory records, and recorded an investigation proposal. It demonstrated no operational recovery.

Challenges and lessons learned

The central challenge was separating plausible advice from verified experience. Model confidence and agreement cannot establish that a historical action succeeded.

Code review also exposed a configuration-scope bug: verification fingerprints included a version that incident storage and retrieval did not preserve. I corrected the complete path, added regression tests, and applied an additive database migration before deploying the updated application.

Infrastructure troubleshooting required separating storage failures from Python errors and network connectivity from database authentication. Moving development to internal storage and testing an alternate hotspot restored progress.

Accomplishments

The completed release includes a public deployment, inspectable evidence replay, downloadable experiment records, and 60 passing local tests.

The repository contains the PostgreSQL migrations, container configuration, AWS infrastructure template, and deployment verification records.

What comes next

Future work will expand the incident set, add independent operational verifiers, and evaluate outcome quality across models. Any remediation executor will require explicit authorization and post-action verification before its results can become trusted memory.

Built With

Share this project:

Updates

Submission history