CrisisOps

Inspiration

Every SRE team we talked to had the same story: the first 15-20 minutes of an incident aren't spent fixing anything -they're spent figuring out what's even wrong, while alerts pile up, Slack fills with "is anyone looking at this?", and the same root cause that took down the service six weeks ago gets rediscovered from scratch because nobody remembers.

We didn't want to build another alerting dashboard. We wanted a system that could reason through an incident the way a senior on-call engineer does -cross-checking logs against metrics against history -and act on that reasoning when it was genuinely confident, instead of just paging a human and hoping. Build Week was the push to actually finish it: we had an existing incident-response prototype sitting half-built, and the challenge to build meaningfully with Codex and QWEN was the reason to bring it the rest of the way.

What it does

CrisisOps is a multi-agent incident-response system powered by Qwen. When an alert fires:

1. A Commander agent classifies severity and routes the incident -P1s get an immediate fast-path notification, everything else goes through full investigation.
2. Specialist agents (logs, metrics, historical memory) investigate in parallel, each reporting a finding with its own confidence score.
3. An adjudication layer reconciles their findings -weighted by each agent's confidence and its historical accuracy -rather than simply averaging or picking the loudest signal.
4. A Triage agent produces a root cause, a confidence score, a blast radius estimate, and checks the finding against a library of approved runbooks.
5. A remediation gate -deliberately kept as deterministic code, not another model call -decides what happens next: high confidence and a low-risk, allowlisted fix auto-executes and self-verifies; anything riskier waits for one-click human approval; anything uncertain just surfaces findings and asks.
6. Every step writes to a full audit trail -every prompt, every model response, every confidence score, every branch taken -so the system's reasoning is inspectable, not a black box. To make this demonstrable rather than theoretical, we built a live, intentionally breakable storefront application that CrisisOps can actually diagnose and repair end-to-end -including real verification and automatic rollback if a fix doesn't take -so the pipeline can be watched working in real time instead of taking our word for it.

How we built it, and where Codex actually helped

We used Codex, in ChatGPT, as the primary hand on the codebase for the work done during Build Week -not just for autocomplete, but for the actual engineering decisions:

• The QWEN migration. Our agent layer previously called a different model provider. We handed Codex the full set of per-agent prompts (Commander, Logs, Metrics, Historical Memory, Adjudication, Triage, Communication, Documentation) along with their required JSON output schemas, and had it rebuild the API client layer against the OpenAI API, model by model -routing the reasoning-heavy agents (Triage, Adjudication) to a stronger QWEN tier and the high-volume, low-complexity agents (Logs, Metrics, Communication) to a faster, cheaper tier. Codex handled the plumbing -request/response shape, retry-on-malformed-JSON logic, token accounting -quickly enough that we spent our own time on the harder question: which agent actually needed the stronger model, and which didn't.
• Finding real bugs, fast. We asked Codex to review the remediation gate logic specifically for correctness and security issues, and it caught two things we'd genuinely missed: an approval step that trusted a client-supplied approver ID with no real verification behind it, and a runbook-matching fallback that silently defaulted to an unrelated runbook when nothing actually matched -a "safe default" that wasn't safe at all. Both got fixed the same session.
• The dashboard rebuild. Our first UI tried to show everything at once and ended up reading as a wall of stat boxes with no clear focal point. We had Codex tear it down and rebuild around two things: a live pipeline flowchart that shows the current step and the branches not taken (grayed out, so a decision is visible, not hidden), and a single scrolling log of exactly what's happening, in order.
• The tool-calling remediation agent. Getting QWEN to reliably pick from a fixed, safe registry of backend tools -rather than inventing actions -took several rounds of prompt iteration with Codex, testing against the live demo storefront until the verify-then-rollback loop behaved correctly on both the success and failure paths. Where we spent our own judgment rather than delegating to Codex: the overall multi-agent architecture and negotiation logic, which agent gets which model tier, and the decision to keep the remediation gate as plain deterministic code rather than another model call -that felt like the one place in the system where we wanted zero non-determinism, since it's the boundary between "the AI suggests" and "the AI acts."

Challenges we ran into

Confidence calibration was harder than it sounds. Early on, every agent reported high confidence regardless of how strong the underlying evidence actually was, which made the remediation gate meaningless — everything looked "safe" to auto-execute. We had to be explicit in each agent's prompt about what should and shouldn't earn high confidence, and build a real gap between "auto-execute" and "needs a human" into the gate logic itself, not just the prompt.

Migrating providers under a deadline is a good stress test. Swapping the model layer mid-project meant re-verifying every agent's output schema still parsed reliably against Qwen's responses, not just assuming prompts written for one model transfer cleanly to another.

Approval security. It's easy to build a human-approval step that looks secure but isn't -catching the client-trusted-approver-ID issue pushed us to think about approval as a real auth boundary, not a UI checkbox.

Making "real-time" actually feel real-time. Our first dashboard attempt was busy without being clear. Rebuilding around a live flowchart plus a single scrolling log -rather than a dozen competing stat cards — was the right call, but it meant throwing away and redoing a full UI pass with limited time left.

What we learned

• Deterministic gates beat model-decided gates. The most important architectural decision we made was keeping remediation authorization as plain code, not an LLM call -auditable, testable, and a judge (or an engineer) can read the exact logic that decided whether something executed.
• Multi-agent negotiation is only as good as its weighting. Fanning a task out to multiple agents and picking the most confident answer isn't enough -weighting by each agent's track record, not just its stated confidence, produced meaningfully better adjudication.
• Codex is most valuable when you already know what "correct" looks like. The sessions where Codex helped most were ones where we could hand it a precise spec (a schema, a security property, a UI structure) and let it execute quickly -not sessions where we were still deciding what the system should do in the first place. That division of labor, us on architecture and judgment calls, Codex on fast, correct execution against a clear spec -is probably the biggest thing we'd carry into the next project.

Built With

• api
• codex
• css
• express.js
• github
• javascript
• json
• mcp
• node.js
• openai
• qwen
• react
• rest
• tailwind