Incident Room

An on-call training simulator: fix a fake outage under time pressure, get coached by NVIDIA Nemotron on Nebius, then stress-test your recovery plan against the condition that breaks it.

Comment

Inspiration

A first production outage can be stressful: noisy evidence, time pressure, and fixes that take minutes to roll out and can make things worse. Runbooks and postmortems help, but they rarely let you practice the moment itself. Incident Room offers a safe place to practice investigating and choosing a mitigation.

What it does

You are on call for checkout at a fictional shop. About one in five checkouts is failing right after a deploy.

  • Investigate and act under time pressure. Metrics, logs, deploy history, feature flags and a runbook. Every look costs a simulated minute, changes take minutes to roll out, and failed checkouts keep counting.
  • Talk it through with a Nemotron teammate. It receives the briefing, the runbook, the current status and your own timeline, never the hidden cause, and it is prompted to coach with questions about evidence and risk.
  • Evidence-linked debrief. The debrief uses recorded simulation data and links the relevant timeline events where available. Nemotron adds a short "Coach's read"; each point must cite those facts, and its numbers are checked against them.
  • Pressure Replay. The same outage, but the release shipped an irreversible database migration, so rolling back makes about 98% of checkouts fail.
  • Stress-test your plan. Write your recovery plan in plain English, for example "if checkout fails after a deploy, roll back". Nemotron is instructed to translate it into simulator steps as written, mistakes included; you inspect and approve the steps. The simulator then runs that exact plan on both versions of the outage. "Your plan worked once. Here is the condition that breaks it." Then revise it and compare the two plans side by side.

How we built it

  • A deterministic simulation decides every outcome. The service state (release, feature flags, timeouts, pods, payment route) drives each minute's metrics and logs, so the same choices always produce the same result. The model never decides whether a fix worked.
  • Nemotron 3.5 Super on Nebius Token Factory does four jobs: teammate, Coach's read, plan translator and plan explainer. Each one is constrained in code (see the next section).
  • FastAPI + plain JavaScript, hosted on one small Fly.io machine. The spending record is SQLite on a persistent volume.
  • A spending guard for a public demo. Every model call reserves its worst-case cost before it is sent; when usage comes back, the real token cost replaces the reservation, otherwise the reservation is kept. The model-call budget is $5 in total, including development spending and a $1 judge reserve; the public share is spread per day until judging ends, with per-visitor limits. If the record is missing, the AI turns off instead of resetting.

How Nemotron and Nebius Token Factory are used

Job What Nemotron does What the code enforces
Teammate Coaches during the shift Receives only the briefing, runbook, status and the trainee's own timeline, never the hidden cause (coaching with questions is a prompt rule, not enforced in code)
Coach's read Interprets the debrief Every point must cite fact IDs; points with unknown IDs, numeric values absent from the cited facts, their referenced events, comparison rows or runbook, or the wrong tone are dropped
Plan translator Turns plain English into simulator steps Instructed to preserve the plan, mistakes included; strict JSON validation before anything runs; unrequested status updates removed in code; assumptions and unsupported parts are shown; the trainee approves
Plan explainer Explains why the plan behaved differently in A and B Same citation and number checks, with facts scoped per version

All calls use the OpenAI-compatible chat/completions endpoint, with thinking turned off for speed (about 0.5–1.2 s per call in our tests). There are no automatic retries; a failed Coach's read allows one manual retry.

Challenges we ran into

  • Hidden reasoning ate the output. With thinking on by default, 589 of 600 tokens went to reasoning and the reply was cut off. enable_thinking: false fixed it, at 0.54 s and 23 tokens.
  • Keeping the AI honest. Our first number check was too strict: it threw away true numbers that came from the timeline. We widened it to the events a fact cites, and we added tone rules.
  • The translator kept "helping". It added a status update the trainee never asked for, even when told not to. We now enforce that in code.
  • A real budget cap. We could not find a documented hard spending cap on Token Factory, so we built a reserve-then-settle ledger.

Accomplishments that we're proud of

  • An AI-assisted training app where the AI cannot change simulation outcomes. Its explanations pass citation-ID, numeric-value and partial tone checks, but can still be wrong, so they are labelled as AI and kept apart from the facts.
  • The stress test shows a genuine lesson that a chatbot alone cannot: in our runs, the same "roll back" plan recovers in one version (764 failed checkouts in 20 simulated minutes) and breaks checkout in the other (14,098). A "read the release notes first" plan survives both.

What we learned

  • Small, strict jobs for the model, with validation in code, beat one big prompt.
  • Determinism makes AI features testable. We could write regression tests for every bug we found.
  • For public demos, spending control is a product feature, not an afterthought.

What's next

More incidents (database failover, certificate expiry, a noisy-neighbour queue), team mode with a shared incident channel, and an "on-call readiness" path for new hires.

Built With

Share this project:

Updates

Submission history