Cortex — Autonomous Multi-Region SRE Agent Swarm

A 6-agent AI swarm that uses CockroachDB as its only memory. Kill a region mid-incident — the swarm steals the lock from another region and finishes the job. No Redis. No checkpoints. Just CRDB.

Comment

Inspiration

Every SRE team has the same nightmare: your incident-response bot is mid-remediation when the region hosting it goes down. All context — agent state, locks, runbook matches, event traces — vanishes. A second bot spins up elsewhere, but it starts from scratch. Worse, both might come back alive simultaneously and apply conflicting fixes.

We asked: what if the agent's memory didn't live in the agent at all? What if CockroachDB — a database designed to survive regional failures — was the memory? Not a cache. Not a checkpoint. The actual, single source of truth that any agent in any region can read, write, lock, and resume from.

That's Cortex.

What it does

Cortex is a 6-agent autonomous SRE swarm that handles production incidents end-to-end — from alert ingestion to postmortem — with CockroachDB Serverless as its only durable memory.

The swarm runs as AWS Lambda functions deployed identically in two regions (us-east-1 and us-west-2). When an alert fires:

  1. Ingest normalizes the alert and creates (or reuses) an incident in CockroachDB
  2. Triage and Runbook agents run in parallel — triage checks for recurrence patterns, runbook performs vector search over CockroachDB VECTOR(384) columns to find relevant prior fixes
  3. Merge combines their findings into an execution plan
  4. Remediation acquires a CockroachDB distributed lock (20s lease, 8s heartbeat), generates an LLM-powered fix via Groq, and applies it with fenced writes
  5. Postmortem generates a report, embeds it as a vector, and stores it — so the next incident benefits from this one's resolution

The signature demo: kill us-east-1 mid-remediation (set Lambda concurrency to 0). The lock heartbeat dies. The lease expires. Fire the same alert at us-west-2 — it reads CockroachDB, sees the expired lock, atomically steals it, and finishes the job. One database. Two regions. Zero data loss.

How we built it

Agent Framework: LangGraph StateGraph with Pregel superstep parallelism for the triage/runbook fan-out. Deliberately no LangGraph checkpointer — the graph's execution state is disposable; CockroachDB is the only durable memory.

Database Layer: CockroachDB Cloud Managed MCP Server for all reads and inserts (select_query, insert_rows), with automatic asyncpg direct-SQL fallback. Per-agent SQL roles with least-privilege RBAC — only remediation_agent can touch incident_locks.

Vector Search: sentence-transformers/all-MiniLM-L6-v2 (384-dim) via fastembed/ONNX, running locally inside Lambda. No external embedding API, no per-token cost. CockroachDB's native CREATE VECTOR INDEX powers cosine-distance retrieval over runbooks and postmortems.

Distributed Locks: CockroachDB serializable transactions power the lock-steal mechanism. The UPDATE ... WHERE lease_expires_at < now() query is race-free by default — no application-level locking needed.

Infrastructure: AWS SAM deploys the exact same template to both regions. SSM Parameter Store holds all secrets. S3 stores raw runbook content. Lambda Function URLs provide the API surface.

Frontend: React 19 + Vite 8 with a custom transit-map SVG visualization that animates the 6-agent DAG in real-time as events stream from CockroachDB.

Challenges we ran into

  • Lambda's read-only filesystem vs. fastembed: The embedding model needs a writable cache directory. Solved by pre-packaging the model as a tar.gz and extracting to /tmp on cold start.
  • MCP Server limitations: The Managed MCP Server only supports SELECT and INSERT, not UPDATE. Built a dual-path architecture — MCP for reads/inserts, direct asyncpg for status transitions and lock mutations.
  • Race conditions in the parallel fan-out: Triage and runbook agents running concurrently could race on status transitions. Solved by making both branches read-only and deferring all writes to the sequential merge node.
  • Lock heartbeat architecture: The heartbeat can't live inside the LangGraph node — a killed process takes any in-graph loop with it. Moved it to the Lambda handler as an asyncio background task with a shared mutable dict side-channel.

Accomplishments that we're proud of

  • The lock-steal actually works. Kill a region on camera, watch the other region steal the lock and finish — not simulated, not mocked, real CockroachDB serializable transactions across real AWS regions.
  • The memory compounds. Every resolved incident generates an embedded postmortem that becomes searchable context for future incidents. Run 5 incidents and the 6th one genuinely finds more relevant fixes.
  • Zero external vector DB. CockroachDB handles relational state, vector search, AND distributed locking in one database. No Pinecone. No Weaviate. No Redis.
  • Per-agent SQL RBAC. Even a compromised triage agent structurally cannot forge a lock — only remediation_agent has grants on incident_locks.

What we learned

  • CockroachDB's serializable isolation makes distributed mutex patterns surprisingly simple — the hardest part was believing the UPDATE WHERE lease_expires_at < now() query really is race-free without extra locking.
  • The Managed MCP Server is a powerful abstraction for agent-database communication. Standardizing on tool calls instead of raw SQL made the agent code dramatically cleaner.
  • LangGraph's Pregel superstep model maps beautifully to SRE workflows — the fan-out/merge topology is a natural fit for "analyze in parallel, act sequentially."

What's next for Cortex

  • Multi-cluster CockroachDB: Leverage CockroachDB's multi-region table localities to pin incident data to the originating region while keeping locks globally consistent.
  • Agent Skills integration: Replace the hand-authored runbook seed with CockroachDB Agent Skills for dynamic, self-updating operational knowledge.
  • Expanded swarm: Add specialized agents for capacity planning, cost estimation, and automated rollback verification.
  • Production hardening: Replace the shared-secret admin key with AWS IAM auth on Function URLs, add rate limiting, and implement proper observability with OpenTelemetry.

Built With

Share this project:

Updates