OpsMemory AI

An agentic AWS SRE assistant that remembers operational experience. It uses CockroachDB vector search to recall past incidents and resolve new ones faster — learning from every fix it makes.

Comment

Inspiration

Every SRE team faces the same problem: when a production incident hits at 3 AM, the on-call engineer often lacks context about what happened last time. Runbooks go stale. Institutional knowledge lives in people's heads. Senior engineers leave, and their experience goes with them.

We asked: what if the incident response system itself could remember — not just store documents, but recall operational experience the way a veteran SRE does? What if it could look at a new incident and say, "I've seen something like this before, and here's exactly what fixed it"?

CockroachDB's distributed vector indexing and MCP Server made this possible: a single, transactionally consistent database that stores incidents, embeddings, agent state, and audit trails — all unified. No separate vector DB. No eventual consistency gaps. One source of truth for the agent's entire memory.

What it does

OpsMemory AI is an agentic AWS SRE assistant with persistent experiential memory powered by CockroachDB. When a new AWS incident occurs, it:

  1. Investigates current AWS telemetry (CloudWatch metrics & Lambda logs)
  2. Searches memory via CockroachDB's distributed vector index to find semantically similar past incidents
  3. Retrieves full historical experience — symptoms, root cause, remediation, and outcome
  4. Reasons about the current incident using Amazon Bedrock Nova Pro, informed by historical context
  5. Recommends a specific remediation — human approval required before any infrastructure change
  6. Executes the approved fix (actually modifies AWS infrastructure)
  7. Stores the new experience as memory — the agent grows smarter with every resolution

The key demo: Historical incident INC-1832 was resolved by increasing SQS visibility timeout. When new incident INC-1947 arrives with completely different wording ("messages reprocessing, queue depth growing"), the vector search finds 86% semantic similarity to INC-1832. Nova Pro diagnoses the same root cause and recommends the same fix — which then actually changes the SQS queue configuration from 60s to 180s.

"OpsMemory doesn't just retrieve documents. It remembers operational experience."

How we built it

Architecture: 6 AWS Lambda functions forming a tool-using agent workflow, connected to CockroachDB Cloud as the unified memory layer and Amazon Bedrock for AI reasoning.

  • CockroachDB Cloud (Basic, AWS us-east-1): Single database storing incidents, 1024-dim vector embeddings (HNSW indexed), agent run state, approvals, audit logs, and remediation history. The MCP client interface provides structured, auditable access.
  • Amazon Bedrock Nova Pro (amazon.nova-pro-v1:0): Core reasoning — receives symptoms, telemetry, and historical matches; produces structured diagnosis with confidence scoring.
  • Amazon Bedrock Titan Embed Text v2: Generates normalized 1024-dimensional embeddings for semantic similarity matching.
  • AWS Lambda (6 functions): Orchestrator (state machine), Telemetry (CloudWatch), Memory Search (vector queries), Reasoning (Nova Pro), Remediation (SQS API), Verification (health checks).
  • API Gateway + S3 + CloudWatch: REST API, frontend hosting, observability.
  • AWS SAM: Infrastructure-as-code deployment.

Key technical decisions:

  • CockroachDB as the SINGLE store — no separate vector DB, no Redis for state, no DynamoDB for sessions. Transactional consistency between incident data and its embedding.
  • Human-in-the-loop for all destructive operations.
  • esbuild for Lambda bundling — sub-second builds, tiny packages.

Challenges we ran into

  1. Lambda execution model: API Gateway terminates Lambda when the HTTP response is sent, killing async background work. Solved by making the investigation workflow synchronous within Lambda's 120s timeout.

  2. API Gateway timeout: The 29s timeout is too short for the full approve→execute→verify chain. Handled via status polling pattern.

  3. Semantic similarity calibration: Finding the right cosine distance threshold — too strict misses valid matches, too loose creates false connections. Settled on 0.5 threshold; our demo achieves 86% similarity.

  4. Nova Pro structured output: Getting reliable JSON from the model required careful prompt engineering with fallback parsing for markdown-wrapped responses.

  5. CockroachDB vector index syntax: The HNSW index creation differs slightly from standard pgvector; iterated to find the correct CREATE INDEX ... USING hnsw (embedding vector_cosine_ops) syntax.

Accomplishments that we're proud of

  • 86% semantic similarity between incidents with completely different wording — proving experiential memory works
  • Actual AWS infrastructure change: The agent changed SQS visibility timeout from 60s → 180s in a real account
  • End-to-end in under 5 seconds: Incident → telemetry → memory search → reasoning → recommendation
  • 12/12 system health checks passing: Every component verified operational
  • Unified memory architecture: CockroachDB stores incidents + embeddings + agent state + audit trail — zero sync issues

What we learned

  • CockroachDB's distributed vector indexing is production-ready and eliminates the need for a separate vector database
  • The MCP protocol provides a clean abstraction for agent-database interaction with built-in auditability
  • Amazon Nova Pro excels at structured reasoning tasks with low temperature settings
  • Persistent experiential memory fundamentally changes agent behavior — they accumulate wisdom rather than starting fresh
  • The "agentic memory" constraint pushed us toward a genuinely more useful design than standard RAG

What's next for OpsMemory AI

  • Multi-tenant memory sharing across SRE teams
  • Memory confidence decay (weight recent experience higher)
  • Automated runbook generation from accumulated experience
  • Proactive incident prediction using memory patterns
  • Extended remediation catalog (Auto Scaling, RDS failover, Lambda config)
  • Real-time CloudWatch alarm → investigation trigger via SNS

Built With

Share this project:

Updates

Submission history