Pager Mutation Score

Mutation testing for your on-call: every merge request proves that each outage it could cause is caught, and a GitLab Duo flow drafts a repair when it isn't.

Comment

Inspiration

On January 31 2017, GitLab.com lost production data. Backups had been failing for a while, and the emails that reported it were rejected, so nobody knew (GitLab's postmortem). The monitoring existed. It had never been tested against the failure that mattered.

Code has mutation testing: break the code on purpose and check that a test notices. Alerting rarely gets the same treatment. Rules and routing are edited by hand, syntax-checked at best, and their first real test is a real outage at 3 AM. We wanted every merge request to answer one question before it merges: if this change breaks something, does the right person hear about it?

What it does

On each merge request, the page-check job starts the real app (Cedar Court, a small dues API) and breaks its one dependency, a holiday calendar, in eight ways while steady user traffic runs: it hangs, returns 503s, slows 1 request in 4 by 700 ms, returns truncated JSON, refuses connections, returns 200 with an error inside, answers 1 in 2 with 503 and retry-after: 0 (inviting a retry storm), and fails 1 call in 100. Plus a quiet day with a healthy calendar.

  1. Record in real time. 60 s quiet, then 6 minutes of fault at 10 user requests per second. The app's own /metrics is scraped every 5 s; a recording whose scrapes drift more than 500 ms is rejected. The load generator records what users actually got.
  2. Evaluate the repo's own rules, unmodified, with Prometheus' engine. promtool test rules replays every recording against monitoring/rules.yml, so rate(), for: and burn-rate windows behave exactly as in Prometheus.
  3. Route every alert that fired through monitoring/alertmanager.yml with amtool.
  4. Score. If users were hurt, a page must reach gitlab-pager within 5 minutes. If the app's fallback held, a ticket must reach gitlab-tickets, and a page is a false page. The quiet day must fire nothing. Every miss is named as a code, rule or routing gap, and the job fails on any miss or false page.

route-check checks that every alert the rules define reaches the receiver its severity needs. page-live sends every alert that fired through a real Alertmanager with the repo's config into a webhook sink inside the job.

Two modeled inputs, both printed on every scorecard: the 1 h and 6 h rule windows need history a 7-minute recording can't have, so each replay starts with 6 hours of modeled quiet traffic at the rate measured in the same run, with zero errors; and the slow-burn row is computed on a modeled 6-hour continuation at the error ratio measured in its recording. Everything else is recorded from the real app.

When a pipeline fails, a GitLab Duo Agent Platform flow with four agents wakes up (trigger: Pipeline events, Failed):

  • Gate reads the pipeline event and hands the flow to the pager agent only when page-check failed on a merge request pipeline. A red unit test never wakes the pager agent, and neither does a failed pipeline on the flow's own pms-repair- branch.
  • Pager reads scorecard.json, the diff, the rules and the routing, points each missed outage at the line behind it, commits one bounded repair on a new branch, opens a Draft merge request, and comments on the original merge request. If the fix is out of bounds (a harness change, a new dependency, a product decision), it files a Monitoring debt: issue instead.
  • Skeptic reviews only the Draft's diff, with no pipeline tools, against fixed rules: no alert deleted or disabled, no trivially low threshold, no for: lengthened to dodge, no edits to the harness or pipeline, no page route without severity="page", and the outbound call and its metrics kept. It posts Skeptic: APPROVE or Skeptic: REJECT.
  • Pages is built to turn each page page-live delivered into a GitLab incident (not run live yet).

The Draft's own pipeline reruns the same mutants with the same harness. That pipeline is the proof, not the agents' word. The flow never merges, approves or enables auto-merge; a person decides.

The live run on GitLab.com (Oct 25 2026, Mountain Time)

Everything here is on public pages of the project. Nothing has been merged.

  • MR !1 (calendar-live into main) switches the next-due calculation to a live holiday calendar. It is a demo merge request: we seeded its monitoring with four gaps on purpose, each marked with a "Seeded gap" comment in the files, and its new calendar client trusts any 200 response. Its pipeline 2988040064 failed at 11:01:13 PM: page-check, route-check and page-live red. The page-check job scored it 0 of 8 outages caught: 4 routing gaps, 3 rule gaps, 1 code gap. Pages did fire for four of the outages, but the pager route matches cedar_court while the alerts carry cedar-court, so all of them landed in blackhole; the page-live job replayed 5 pages and delivered 0.
  • The flow woke up. At 11:01:15 PM, two seconds later, workload pipeline 2988067872 started as the flow's service account and ran for 3 minutes.
  • The repair. The flow created branch pms-repair-1 and pushed commit 932a96f2, authored by the service account: the pager route matcher, the latency rule's route selector, a dependency ticket rule's for: cut from 15m to 2m, a new slow burn-rate tier, and a calendar client with a timeout, a body check, a snapshot fallback and a metric on every attempt. It opened Draft MR !2 into calendar-live and posted a table on !1 with each missed outage, the line behind it and the fix. GitLab lists lume grid (lumegridai) as the author of !2; its commit and pipelines are by the service account.
  • The skeptic. On !2, the flow posted Skeptic: APPROVE with all six checks (R1 to R6) passing, read from the diff only.
  • The proof. !2's pipeline 2988078421 passed every job. Its page-check job scored 8 of 8 caught (2 paged, 5 ticketed, 1 absorbed with nobody woken), 0 false pages, with the same harness hash as !1's run (a1cc7900494b, taken from the target branch). Its page-live job delivered 2 pages to gitlab-pager and 6 tickets to gitlab-tickets through a real Alertmanager, 0 to nobody.

The flow is public in the AI Catalog: https://gitlab.com/explore/ai-catalog/flows/1019767. Its YAML is also in the repo at .gitlab/duo/flows/pager-mutation-score.yaml.

How we built it

  • GitLab Duo Agent Platform: one custom flow, four agent components (gate, pager, skeptic, pages) joined by a router on the gate's one-word answer. Trigger: Pipeline events (Failed) plus Mention. Tools come from GitLab's tool registry only; the skeptic gets no pipeline tools. scripts/validate-flow.py checks the YAML against GitLab's own flow_v2.json schema, then checks every router target, prompt input and tool name.
  • GitLab CI/CD: test, page-check, route-check, page-live, GitLab's SAST and Secret Detection templates, and a release job that publishes the scorecard on a tag (built, not yet run on a tag). Scorecards (.svg, .md, .json) and pages.json are exposed as merge request artifacts.
  • Prometheus 3.15.1 and Alertmanager 0.34.1 from the official release tarballs, checked against pinned SHA-256 sums in the job. promtool evaluates the rules; amtool routes; a real Alertmanager delivers in page-live.
  • Node.js 20+, node:test, no runtime dependencies beyond acorn. The pipeline needs no paid services, no CI variables and no alert integration.
  • A hash-locked harness. On a merge request, page-check runs the target branch's harness (mutants, expectations, recorder, scorer) against the merge request's app and monitoring, and fails if the merge request changed the harness. CODEOWNERS guards the pipeline itself once code owner approval is switched on.

GitLab lifecycle stages that ran live on this project: Create (the repair on Draft !2), Verify (page-check and route-check red on !1, green on !2), Configure (alert rules and routing as code, evaluated by Prometheus' own tools), Monitor (page-live delivering through a real Alertmanager, green on !2) and Govern (the hash-locked harness, SAST, Secret Detection, and the skeptic's review). Built but not yet run live: Plan (Monitoring debt: issues), Release (scorecard on a tag) and incidents from the pages agent.

Challenges we ran into

  • Honest time. Burn-rate rules with for: clauses only mean something on real timing, so the recordings run in real time and promtool replays them at the same 5 s cadence. When a busy machine stalled twice for 13 to 20 s during a local run, every recording had a scrape too far from its slot, and page-check refused to score all 36 instead of guessing.
  • History we can't record. 1 h and 6 h windows need hours of traffic. We model a quiet lead-in from the same run's measured request rate, with zero errors, and print that on every scorecard rather than hide it.
  • The trigger's Instructions field. With instructions set, the flow receives them plus a short context block instead of the pipeline event, so the gate has nothing to read. Leaving Instructions empty hands the flow the raw pipeline event.
  • The flow schema. The flow editor validates against a stricter schema than the prose docs: no top-level name or description, no max_cycles on agents. Each prompt ends with its own stop rule, and we validate locally before pasting.
  • Developer access only. We couldn't add an alert integration or masked variables, so page-live runs a real Alertmanager into a webhook sink inside the job, guarded by a Bearer key generated in that job. It never claims GitLab received a page.

Accomplishments that we're proud of

  • A real merge request on GitLab.com went red for the right reason; two seconds later a flow woke up and wrote a repair that a second agent reviewed. !1's pipeline scored 0 of 8; the repair's own pipeline scored 8 of 8 with the same harness. Nothing was merged.
  • Every alert time and route in the scorecard comes from promtool and amtool, every first-alert time is asserted back to promtool, and the saved test files replay with promtool test rules alone.
  • The scorecard names each miss as a code, rule or routing gap, and the pager agent pointed each one at its line (the four monitoring gaps are labeled "Seeded gap" in the files; the client gap is not): a pager route typo, a rule on a label the app never emits, a for: that outlives the deadline, a missing burn-rate tier, and a client that trusts a 200 with an error inside.

What we learned

Alerting fails quietly and in layers. In the bad merge request, alerts did fire for four of the outages, and the pages still reached nobody because of one character in a route. Rule tests alone would have passed. route-check catches that typo in seconds, but only the full path, from real metrics to a delivered page, shows which outages users would have felt with nobody paged.

What's next

  • Run the pages agent and Monitoring debt: issues live, and publish the scorecard as a release on a tag.
  • More dependencies and mutant shapes (databases, queues), and a forward to a real alert endpoint by default.
  • Score a project's existing alerting against its real traffic shape instead of a modeled quiet lead-in.

Prior art

promtool unit tests use hand-written series; chaos tools can check a monitor during an experiment. Pager Mutation Score records series from the real app under real faults, scores them on every merge request, and pairs an agent that repairs the gap with a second agent that checks the repair.

Try it

Node 20+, git, curl and sha256sum. One command scores the demo merge request and its repair, in parallel, in about 8 minutes:

npm ci --ignore-scripts && npm run demo

To use it on your own GitLab project: push the repository, add the flow (AI > Flows > New flow, or from the AI Catalog), then add a flow trigger with Pipeline events (Failed) and Mention, leaving Instructions empty. Steps are in the README.

Built With

Share this project:

Updates

Submission history