SRE Copilot: Evidence-Driven Incident Triage

AI-powered SRE incident triage with evidence-driven RCA and human-approved remediation.

Comment

Inspiration

Incident response often breaks down because evidence is scattered across alerts, logs, traces, metrics, change history, and runbooks. We wanted to explore how an AI SRE copilot could help engineers reason faster during incidents while still keeping production-changing actions behind explicit human approval.

What it does

SRE Copilot is a public-safe, synthetic demo for evidence-driven incident triage. It turns mock observability data into an auditable RCA workflow with:

  • multi-agent investigation across a production-style topology
  • three synthetic incident scenarios
  • explainable confidence scoring
  • read-only runbook retrieval
  • copyable RCA briefs
  • local-only approval packets for blocked remediation actions
  • a clear boundary between AI recommendations and human-approved operations

How we built it

We built the demo as a static web app using HTML, CSS, JavaScript, and Three.js for the animated RCA topology. All incident data, runbook snippets, logs, traces, metrics, and changes are synthetic.

Codex helped us design the architecture, implement the UI, refactor the scenario data model, build the RCA simulation workflow, run reviews, improve accessibility, and deploy the project to GitHub Pages.

Challenges we ran into

The biggest challenge was balancing a compelling AI-agent demo with operational safety. We did not want the project to imply that an AI system should autonomously restart services, roll back changes, or notify customers. The design therefore separates read-only evidence gathering from approval-gated actions.

Another challenge was making the demo understandable quickly for judges while still showing realistic SRE details like traces, logs, data gaps, confidence limits, and runbook context.

Accomplishments that we're proud of

We are proud that the project demonstrates an end-to-end SRE workflow without relying on private data or credentials. The demo includes a cinematic topology view, parallel and serial agent stages, scenario switching, RCA brief generation, explainable confidence, and local-only approval packet generation.

We are also proud of the safety model: every production-changing action is clearly marked as blocked until a human approves it.

What we learned

We learned that the most valuable AI behavior in incident response is not just summarization. It is evidence organization, uncertainty tracking, and clear handoff generation.

We also learned that Codex is effective for rapidly turning a product idea into a working, reviewed, deployable demo, especially when the data model, UI, documentation, and validation scripts all need to stay aligned.

What's next for SRE Copilot: Evidence-Driven Incident Triage

Next, we would add a real model adapter with structured outputs, stronger validation around generated RCA artifacts, and integrations with approved observability and incident-management tools.

We would also expand the approval workflow so teams can route mitigation requests to the right human reviewer while preserving a complete audit trail.

Built With

Share this project:

Updates