Distributed AI SRE Assistant with Agentic Memory

pre-demo run
Demo run 2
Demo run 1

SRE teams are overwhelmed by alerts and lose institutional knowledge when staff members rotate. Every AI assistant today has amnesia; it cannot remember what worked last time. We observed on-call engineers re-diagnosing the same class of incidents repeatedly, losing 30-60 minutes each time because the resolution from the previous month lived in someone's head, not in a system. We wanted to build an agent that genuinely learns, one that resolves incident #2 faster than incident #1 because it remembers what worked, stored in a database that never goes down, even if an entire AWS region disappears. What we learned • CockroachDB's multi-region GLOBAL tables with SURVIVE REGION FAILURE provide the strongest durability guarantee we've seen in a distributed database, writes aren't acknowledged until replicated across 3 regions via Raft consensus • Titan Embeddings V2 returns 1024-dimensional vectors that perform just as well as larger dimensions for incident similarity matching • The "memory improvement delta" is real and measurable: our LLM-as-Judge scored agents +0.60 higher (on a 5-point scale) when memory was available vs cold starts • Claude Sonnet 4.5's chain-of-thought reasoning genuinely changes when you inject prior resolution context into the prompt, it cites the prior incident and boosts its confidence accordingly • Building evaluation into the system from day one (not as an afterthought) made every architectural decision easier to validate How we built it Memory Layer (CockroachDB): 3-region cluster (us-east-1, us-east-2, us-west-2) with SURVIVE REGION FAILURE. Four GLOBAL tables: incidents, incident_embeddings (vector index with cosine similarity), agent_memory (reasoning chains), and agent_decisions (with LLM-as-Judge scores). Every agent write is synchronously replicated across all regions before the acknowledgment. AI Layer (AWS Bedrock): Claude Sonnet 4.5 for chain-of-thought root-cause analysis and remediation planning. Titan Embeddings V2 for 1024-dim incident vectors. The Analysis Agent's prompt explicitly changes its behavior when prior resolutions exist; this is the mechanism that produces measurable improvement. Orchestration: AWS Step Functions state machine with five Lambda steps (ingest → retrieval → analysis → decision → memory write). Human-in-the-loop approval gate using task tokens. Error handling preserves the partial state of memory so that knowledge is not lost during a crash. Evaluation: LLM-as-Judge scoring across four dimensions (accuracy, resolution quality, reasoning, and evidence usage). Paired evaluation runner that fires incident A (no memory), resolves it, then fires A' (same class), measuring confidence delta, reasoning step reduction, and whether the prior resolution was cited. Frontend: Streamlit three-panel UI showing alert signals, agent reasoning stream with color-coded steps, and memory recall panel. Performance comparison chart showing the improvements between the runs. Challenges we faced • Vector dimension mismatch: Designed for 1536-dim embeddings (Titan docs), but the API returned 1024. Required schema migration mid-build and vector index recreation across three regions. • Model retirement during development: It was discovered that 3.5 Sonnet was retired on bedrock and perhaps at large between our design phase and implementation. Migrated to Claude Sonnet 4.5 with updated model IDs and API patterns. • Evaluation honesty vs. demo appeal: Our n=5 paired evaluation showed +0.60 average improvement, but only 2/5 pairs strictly improved (ceiling effect, baseline was already 4.15/5.0). I chose to document the limitations honestly rather than cherry-pick the results. • Multi-region cost management: A standard plan with three regions under a $400 credit budget required careful capacity planning (2 vCPUs, usage-based storage). • Same-model judge bias: Claude scoring Claude's output is a known limitation. We documented this and noted production mitigation (cross-model judgment or human evaluators). What's next for SRE Memory Agent • Deterministic guardrails layer: Add prompt injection detection, PII masking, and RBAC policy enforcement before any LLM call reaches the agent pipeline • Cross-model judge evaluation: Replace same-model scoring with a different model family (or human evaluators) to eliminate self-bias in the LLM-as-Judge pipeline • Expand to 50+ incident classes: Scale the evaluation dataset with the Bedrock-powered variant generator to produce statistically significant improvement metrics with confidence intervals • Real CloudWatch integration: Replace synthetic alerts with live CloudWatch alarm triggers, making the system production-deployable for actual SRE teams • OpenTelemetry distributed tracing: Instrument all agent steps with OTel spans for enterprise-grade observability and compliance audit trails • Memory decay and relevance ranking: Implement time-weighted memory so recent resolutions rank higher than stale ones, preventing outdated fixes from being applied to evolved infrastructure • Multi-tenant memory isolation: Extend the CockroachDB schema with tenant-scoped access so multiple teams can share the platform without leaking incident context across boundaries • Autonomous remediation execution: Graduate from "propose and approve" to safe automated execution with rollback verification, closing the loop from diagnosis to resolution without human intervention