War Room: AI Incident Response

Inspiration

When production breaks at 3 AM, the first hour is lost to archaeology: what changed, which deploy, whose commit? On-call engineers dig through logs and git history under pressure while customers stare at error pages. We wanted to see whether a team of AI agents could run that entire loop — detect, diagnose, fix, verify, document — and do it transparently enough that an engineer would actually trust it.

What it does

War Room is an AI incident-response team with every agent's reasoning visible live. Press one button to deploy a real regression into a live demo service — the system literally checks out a guilty commit and restarts the service. Real traffic starts failing, an error-rate alert fires from actual structured logs, and five GPT-5.6 agents respond in sequence:

• Triage sets severity and an investigation plan
• Log Detective reads the real stack traces and ranks hypotheses with quoted evidence
• Git Detective receives the deploy window's actual diffs — the guilty commit buried among innocent ones — and names the SHA from evidence alone
• Fixer writes the patch, commits it to a fix branch, redeploys, and proves recovery: traffic replays and the error rate drops from ~100% to ~5% on screen
• Scribe writes the postmortem from the incident timeline

Three planted regressions ship with the repo (connection-pool leak, null dereference, timeout typo), each isolated on its own branch so every outage has exactly one root cause. A keyless demo mode runs the whole loop with zero credentials, so anyone can test it in two commands.

How we built it

The core was built with Codex running GPT-5.6 across iterative sessions: the pnpm monorepo, the SSE streaming bus, the seeded git repository with authored guilty commits, the deploy-by-checkout fault injector, the five-agent pipeline, the fix/redeploy/measure loop, and the React dashboard. At runtime, GPT-5.6 also powers the product itself — all five agents call it through the OpenAI Responses API with token-by-token streaming into the war-room feed.

Challenges we ran into

The hardest problem was honesty. Our first fault injector just wrote a fake alert — the "investigation" was rigged, and the alert text contained the answer. We rebuilt it three times until nothing was staged: faults became real code deployed via git checkout, alerts became thresholds over genuinely failing traffic, and the Git Detective's evidence window became a multi-commit range where the guilty commit hides among innocent ones. Other battles: a seeding race where incidents fired before the demo repo finished generating, pnpm module-resolution breaking spawned child services (solved by making the demo service dependency-free), and scenario isolation — stacked bugs on one branch made every outage show the wrong symptom.

What we learned

An AI fix nobody can inspect is worthless in incident response — visibility is the product. Streaming every agent's reasoning, showing the fix diff, and measuring recovery with real traffic turned "trust me" into "watch it work." We also learned that demo infrastructure deserves the same rigor as the product: a rigged demo is obvious in thirty seconds, and a real one is what makes the result convincing.

What's next

Real-world connectors (Datadog/PagerDuty alerts in, GitHub PRs out), parallel hypothesis testing across agents, and a replay library of past incidents so teams can train the war room on their own outage history.

Built With

• codex
• express.js
• git
• gpt-5.6
• javascript
• node.js
• openai
• react
• server-sent-events
• typescript
• vite