IncidentTwin

IncidentTwin is an autonomous SRE agent swarm that resolves cloud outages as they are logged using ServiceNow, Amazon Bedrock AI, zero-loss state on CockroachDB, and continuous learning.

Comment

Inspiration

Modern Site Reliability Engineering (SRE) is plagued by alert fatigue, fragmented telemetry silos, and stale runbooks during high-severity outages. When on-call engineers attempt to automate incident triage with AI agents, traditional in-memory agent frameworks introduce a fatal flaw: state loss on container crashes or node failovers.

We were inspired to build IncidentTwin—an autonomous SRE Command Center that pairs ** Amazon Bedrock reasoning with CockroachDB Cloud’s distributed ACID resilience, ensuring AI agents never lose state mid-investigation and never repeat past remediation mistakes **.

What it does

IncidentTwin autonomously detects, investigates, and mitigates cloud microservice incidents in under 30 seconds:

** Autonomous Multi-Agent Swarm *: 6 specialized agents collaborate to ingest OpenTelemetry MELT streams, traverse CMDB service dependency graphs, and formulate mitigation runbooks. * Zero-Loss Durable State *: Every agent step is transactionally checkpointed in CockroachDB. If an agent worker is hard-killed (kill -9), the restarted worker seamlessly resumes with zero lost context. * Failure-Aware Hybrid Vector RAG *: Embeds anomalous telemetry into 1536-dimensional vector space using Amazon Bedrock, retrieves historical post-mortems via pgvector, and actively filters out failed remediation paths. * 1-Click Human-in-the-Loop Gateway *: SREs review synthesized root causes, confidence scores, and blast-radius impact maps before 1-click execution and automated ServiceNow ticket sync. * Conversational NL2SQL Query Assistant **: Enables engineers to inspect live cluster health and incident history in plain English.

How we built it

AI Reasoning & Embeddings: Amazon Bedrock foundation models for multi-agent reasoning, root-cause synthesis, and 1536-dim embeddings. Distributed Database & Vector Memory: CockroachDB Cloud Serverless (pgvector for SOP retrieval, multi-region ACID transactions for workflow checkpoints, CMDB service graphs, and managed MCP tooling). Backend Orchestration: Python 3.13, FastAPI, LangGraph (cyclical agent DAG), asyncpg, and bidirectional WebSockets. Frontend Command Center: React 18, TypeScript, Vite, React Flow (topological blast radius map), and Apache ECharts (synchronized MELT time-series stream). Cloud Infrastructure: AWS App Runner (unified container hosting with managed TLS & auto-scaling), Amazon ECR, CloudWatch, and Amazon S3.

Challenges we ran into

** State Persistence Across Hard Process Crashes *: Standard LangGraph memory stores state in volatile RAM. We designed an atomic checkpointing engine backed by CockroachDB transactions to guarantee idempotent recovery at any DAG execution node. * Preventing AI Hallucination & Repeat Outages *: Traditional RAG systems often retrieve irrelevant or historically failed playbooks. We engineered negative vector filtering to exclude past failed mitigations based on explicit outcome scoring. * Unified Single-Port Container Packaging **: Bundling the React 18 SPA, FastAPI orchestrator, and demo microservices into a lightweight linux/amd64 multi-stage container that deploys to AWS App Runner with 1 click.

Accomplishments that we're proud of

  • Sub-30s Mean Time to Triage (MTTT) from initial metric anomaly to synthesized remediation plan. 100% Crash Resilience: Proved zero state loss under live chaos testing by killing agent workers mid-approval.
  • Production Live Cloud Deployment: Fully live and publicly accessible on AWS App Runner with enterprise TLS.
  • True Multi-Agent Collaboration: Built 6 distinct, purpose-driven agents operating on a shared distributed memory fabric.

What we learned

** CockroachDB as an Agentic Brain *: CockroachDB is uniquely suited for autonomous AI systems because transactional ACID guarantees and high-dimensional pgvector search reside in the exact same distributed engine—eliminating split-brain memory bugs. * Safe Human-in-the-Loop Guardrails **: Autonomous agents gain enterprise trust when paired with durable approval gateways and clear blast-radius visualization.

What's next for IncidentTwin

** Predictive Incident Forecasting *: Leveraging seasonal time-series foundation models to remediate memory leaks and connection pool saturation before SLA breaches occur. * Multi-Cloud Kubernetes Operator *: Deploying IncidentTwin as a native Kubernetes CRD with automatic eBPF network trace auto-discovery. * Autonomous Safe Rollbacks **: Direct integration with AWS CodeDeploy and ArgoCD for zero-downtime canary rollbacks driven by agent consensus.

Built With

Share this project:

Updates