Resilix SRE — Autonomous Self-Healing Agent

Inspiration

AI agents are rapidly moving from experiments into real production DevOps workflows — running pipelines, diagnosing outages, and managing infrastructure. However, current agentic architectures suffer from a critical flaw: Agent Amnesia.

When a database transaction deadlock or latency spike occurs, traditional AI chatbots analyze logs from scratch. They have no long-term memory of past post-mortems, fail to track transactional state across container restarts, and cannot safely audit live cluster telemetry. An agent whose memory goes offline doesn't degrade gracefully — it stops.

We built Resilix SRE to solve this. CockroachDB serves as the ultimate system of record for agentic memory: globally distributed, always-on, PostgreSQL-compatible, and natively unifying transactional SQL state with distributed HNSW vector embeddings.

What it does

Resilix SRE is an autonomous self-healing SRE agent powered by CockroachDB and AWS Bedrock that continuously monitors, diagnoses, and mitigates production incidents with persistent memory:

1. Triggers Incident Detection: Captures real-time microservice latency spikes and row-level database deadlocks.
2. Queries CockroachDB Vector Memory Store: Executes native HNSW cosine similarity search on 1536-dimensional Titan embeddings stored directly in CockroachDB to retrieve historical post-mortems in <15ms.
3. Audits Telemetry via Managed MCP Server: Interrogates crdb_internal system views using CockroachDB's Cloud Managed MCP Server (https://cockroachlabs.cloud/mcp) without custom proxies.
4. Executes CockroachDB Agent Skills: Applies machine-executable database expertise (e.g., lock contention diagnosis, statement fingerprinting).
5. Synthesizes via Amazon Bedrock: Uses Claude 3.5 Sonnet on AWS Bedrock to formulate a safe remediation plan with Human-in-the-Loop SRE approval.
6. Persists New Incident Memory: Automatically embeds and writes resolved incident post-mortems back to CockroachDB, recording an immutable ACID audit log.

How we built it

• Persistent Memory Layer: CockroachDB (v23.2 / Serverless) with VECTOR(1536) extension and USING HNSW (embedding vector_cosine_ops) index.
• AI & Embedding Models: Amazon Bedrock (anthropic.claude-3-5-sonnet-20240620-v1:0 for reasoning, amazon.titan-embed-text-v1 for 1536d vector generation).
• Backend Service: FastAPI (Python 3.12) with asynchronous ReAct agent orchestrator, psycopg3 PostgreSQL driver, and resilient fallback engine.
• Frontend Dashboard: React 19, TypeScript, Vite, Tailwind CSS with Light/Dark theme switching, real-time SRE terminal console, and Human-in-the-Loop operator modal.

Challenges we ran into

• Unifying Vector & Transactional State: Designing a clean database schema combining high-dimensional vector similarity queries with ACID status transitions (TRIGGERED → ANALYZING → PENDING_APPROVAL → RESOLVED).
• Resilient Fallback Engineering: Ensuring the application operates with 100% uptime even if cloud endpoints or database credentials are unavailable.

Accomplishments that we're proud of

• Achieving sub-15ms vector retrieval times for historical incident post-mortems.
• Seamless integration of all 3 CockroachDB AI ecosystem tools (Vector Indexing, Managed MCP Server, Agent Skills).
• Building an intuitive, production-grade SRE control dashboard.

What we learned

• How native vector search inside CockroachDB eliminates the need for separate vector databases while maintaining full ACID guarantees.
• How the Model Context Protocol (MCP) streamlines secure cluster telemetry retrieval for LLM agents.

What's next for Resilix SRE — Autonomous Self-Healing Agent

• Multi-region automated failover orchestration using CockroachDB multi-region tables.
• Integration with Kubernetes operators for automatic pod restarts based on persistent agentic memory.

Built With

• amazon-bedrock
• amazon-web-services
• cockroachdb
• fastapi
• mcp
• python
• react
• typescript
• vector-database
• vite