Incident Rehearsal

A fire drill for outages, inside Confluence and Jira: rehearse an incident from your own runbooks, then file the evidence.

Comment

Inspiration

PCI DSS requirement 12.10.2 says the incident response plan must be tested at least every 12 months. Most payment teams write runbooks and postmortems and never run them. When the auditor asks for proof of a test, there is usually nothing to show.

We built this for two people:

  • On-call leads and SREs at payment companies, who want to find out a runbook is stale in a practice run, not during a real card-data incident.
  • PCI program owners, who need a record of what was tested, by whom, and what was found.

What it does

Readiness board. The app scans every runbook and postmortem the signed-in user can see. It checks runbooks for tested, reviewed and has content, and postmortems for complete (unfilled template sections are detected). It shows one score with the math beside it, for example "4 of 15 checks passed".

Exercise room. Pick a drill and it runs on a clock, under a purple EXERCISE band so nobody mistakes it for a live incident. Injects come from your own pages: the incident start from a postmortem, customer reports from related postmortems, a monitoring alert from the runbook's own tools, a capacity signal from the runbook's documented limits. Stakeholder questions arrive with their own deadlines. Paging is simulated and every post is prefixed [DRILL].

Judged answers. Five fixed objectives: acknowledge, set severity, page a named owner, answer stakeholders, open the plan. Every answer option has a verdict (sound or poor) with a reason. Choosing "keep watching" at three times the runbook's documented capacity is a poor call, not a pass. Poor calls get a consequence inject later in the drill.

Record. Press Create record and the app writes:

  • an exercise record page in Confluence, as a child of the source page, labelled exercise-record;
  • Jira tasks for the gaps. A page defect that already has an open task is linked, not filed twice.

The record lists objectives with targets and actuals, your answers with verdicts, the plan version exercised, the PCI DSS 12.10.1 elements this run touched, and a timeline. A different person must sign it off. It says plainly that one exercise is partial evidence, not a full annual test.

Demo pace. A 3-minute version of the same drill, labelled "not evidence", so a reviewer can see the whole flow quickly.

Rovo "Incident Game Master". Ask it how ready you are, why the score is what it is, or what to rehearse next. Or ask it to run a drill: it starts the same exercise the app uses, narrates each inject, presents the stakeholder options as they are, records your decisions and files the record. Chat and app share one exercise, so you can switch between them mid-drill.

MCP tools. Six tools are exposed through rovo:mcp: readiness, explain-score, list-scenarios, latest record, current drill, and record-past-exercise. An outside AI client can ask "which runbooks are untested?" or log a real exercise that happened elsewhere. That last tool only creates a claim. The claim stays pending until a person with edit rights presses Confirm in the app.

Empty site. If the site has no incident pages, the board offers a built-in "Suspected card-data compromise" drill and a button that creates sample pages (an incident response plan, a runbook and a postmortem) in a space you pick.

How we built it

  • Forge Custom UI in a confluence:globalPage, with resolvers for the backend. React front end.
  • Rovo agent (rovo:agent) with 9 actions: get-readiness, explain-score, list-scenarios, prepare-exercise, get-latest-record, start-exercise, get-drill, drill-action, file-record.
  • Rovo MCP server (rovo:mcp) exposing six tools, one of them a write.
  • Scheduled trigger that rescans weekly and keeps a score history.
  • Forge KVS for exercise state, records, review attestations and settings. No data leaves Atlassian; the app makes no external calls.
  • Product reads as the user (asUser), so people only see pages they already have access to. Record pages are written with the Confluence v2 pages API. Jira tasks go through the Jira REST API.
  • Privacy reporting API (report:personal-data) runs weekly. Data for closed accounts is erased.
  • One pure scoring engine shared by the backend and by a UI mock mode. The same code scores a run in Forge and in the Playwright tests.
  • Tests: 274 unit tests on fixtures taken from a real stock Atlassian demo site, and 46 Playwright end-to-end tests on mock mode (light and dark, three viewport widths, axe accessibility checks).
  • Reviews: four security review rounds (the last one: 0 High, 0 Medium open), four first-time-user test rounds with live sessions, and judge-agent rounds that scored the app against the contest criteria.

Challenges we ran into

  • Retired endpoints. Several Confluence v1 routes answer 410 Gone for apps. Creating a page, listing spaces and others had to move to the v2 pages API, granular scopes, or CQL search.
  • Auth that fails quietly. Some v1 routes return NEEDS_AUTHENTICATION when called as the user. Single-page reads and user search now go through content search with the scopes the app already has.
  • invoke returns {} for null. A resolver that returns null reached the UI as an empty object and crashed a tab. The client now normalises nullable responses.
  • Iframe auto-height. The Custom UI iframe resizes to its content, which broke sticky and fixed elements (the clock band and the "Reply needed" bar) and made dialogs jump. We kept the live area short and anchored dialogs to the control that opened them.
  • Search-index lag. After the app creates sample pages, Confluence search does not find them right away. The app waits within a time budget before it relies on them.
  • Honest scoring. A drill you can finish early is a drill you can fake. Ending before the last question is marked "Ended early", demo pace never counts, and only a real-pace run that reached the last question counts as a test.
  • Prompt injection. Injects quote text from Confluence pages, and an agent can write. Page text reaches the Rovo agent and MCP clients as quoted data only. Agent and MCP writes of past exercises are claims that wait for a human to confirm in the app, and chat filing goes only to the saved Jira project.

Accomplishments that we're proud of

  • A complete loop on a live Confluence and Jira site: board, drill, judged answers, record in Confluence, tasks in Jira, with no duplicate tasks on a second run.
  • A Rovo agent that runs a full drill in chat on the same server-side exercise as the app.
  • Evidence rules that hold up: sign-off by someone other than the runner, earned PCI element coverage only for what the run touched, and a plan tile that shows fail, partial or pass with the requirement.
  • Four security rounds with every High and Medium fixed, and four rounds of first-time-user testing where each major finding was fixed.

What we learned

  • Platform behaviour changes under you. Several routes the docs still describe answer 410 for apps, so each live call had to be checked on a real site.
  • Treat every write from an agent or MCP client as a request, not a fact. The pending-claim pattern cost one extra click and removed a whole class of problem.
  • Users trust a score only when they can see the math. Showing "4 of 15 checks passed" next to the number mattered more than the number.
  • A drill is only useful if it can say "that was a poor call, and here is why".

What's next for Incident Rehearsal

  • A Teamwork Graph connector so runbooks from other tools show up on the board.
  • Multi-person drills, where several people play the roles in one exercise.
  • A PDF evidence pack for auditors.
  • A Marketplace listing.

Built With

Share this project:

Updates

Submission history