-
-
Nightjar: rolled back 1 min 17 s after the failed probe, fix verified in production after 13 min 45 s, humans woken up: 0.
-
How an incident runs: probe, rollback, incident, Duo fix, proof, risk gate, staging, production, postmortem.
-
Nine lifecycle stages mapped to real jobs, files and GitLab features in one project.
-
The gate proves the fix: the test fails on the base commit and passes with the fix, so MR !3 (score 16) merges on its own.
-
Written by the pipeline: mitigated in 1 min 17 s, fixed in 13 min 45 s (postmortem on incident #3).
-
The gate says no: MR !4 touches app/tax_rates.py, which Finance owns, so it does not merge and asks a human reviewer.
-
One pipeline: scan, package, staging, production, release. Rollback is skipped because nothing failed.
-
The run on 26 Oct 2026 (UTC): probe red 15:55:31, verified in production 16:09:16.
-
Architecture: GitLab runs the loop, Cloud Run (europe-west9) hosts the app, keyless via Workload Identity Federation.
-
What the gate decides: auto-merge, ask a human reviewer, or fail the job (block). One policy file sets the thresholds.
-
A second live run at night: different bug, rolled back 1 min 15 s after detection, fixed after 12 min 48 s, 0 humans woken up.
What it does
Nightjar handles a production incident on GitLab, from the failed check to the postmortem. In a live run on 26 Oct 2026 it rolled production back 1 min 17 s after the failed probe, and the fix was verified in production 13 min 45 s after it. Nightjar is for small teams that ship daily but have no on-call rotation.
- Monitor. A scheduled probe checks customer journeys in production every 15 minutes. On a failure the pipeline rolls back first, opens an incident with the failing request, expected vs. actual response and a suspected file, and assigns it to the GitLab Duo Developer Flow.
- Fix. Following
AGENTS.md, Duo opens a merge request with the smallest fix and a regression test. - Prove. The new test must fail on the merge request's base commit and pass with the fix; no existing test may change. SAST, secret and container scans run.
- Gate. A risk gate scores the change against a policy file (points per changed line and file, more for dependencies, new findings or a missing proof). At 30 or less, a proven change merges on its own. A change to a protected path, such as the Finance-owned tax rates in CODEOWNERS, is not merged: the gate asks a human reviewer and requests our custom Duo flow, Nightjar Approval Brief, which posts a decision brief. Secrets or edited tests fail the gate (implemented and tested, not triggered in the run). The gate uses the tooling of the target branch, so a merge request cannot loosen its own gate.
- Release. A no-traffic staging revision is smoke-tested, and exactly that revision is promoted. The pipeline writes a postmortem with time to mitigate and time to fix into the incident.
Two live runs (26 and 27 Oct 2026, UTC)
To stage the incident, a human merged cleanup MR !2 although the gate had escalated it (no new test). It removed a strip(). At 15:55:31 the probe failed: a coupon pasted with a trailing space got HTTP 422. Rollback and incident #3 followed in 1 min 17 s. Duo opened MR !3 at 15:59, the gate scored it 16, it merged at 16:02, and the fix was verified in production at 16:09:16. Time to fix: 13 min 45 s. Humans woken up: 0.
Actions show "Niklas Wichter" because the pipeline uses a personal access token; no human acted after the failed probe, and MR !3 was merged by the Duo service account. MR !4 (a VAT rate in app/tax_rates.py, score 24) was escalated, not merged, and is still open; the Approval Brief posted within about a minute.
Evidence: incident #3 (/-/issues/3), MR !3, MR !4, pipeline 2990603466; full table in the README.
A second live run in the night of 27 Oct 2026 (UTC) used a different bug: after cleanup MR !6 removed country.upper() (merged by a human, as above), the probe quote-lowercase-country failed, production was rolled back, and incident #5 opened 1 min 15 s after detection. Duo's MR !7 with two new tests was auto-merged by the gate, and the fix was verified in production 12 min 48 s after detection; humans woken up: 0. Evidence: incident #5 (/-/issues/5), MR !7.
How we built it
- GitLab Duo Agent Platform: custom flow Nightjar Approval Brief (two components: the first reads the merge request with read-only tools, the second can only post one merge request note), the Developer Flow (
AGENTS.md,.gitlab/duo/agent-config.yml) and the Fix CI/CD Pipeline flow (MR !1). - GitLab CI/CD: scheduled and MR pipelines, environments, merge-when-pipeline-succeeds bound to the checked commit, releases, scanners; Python tools in
ci/; a small FastAPI demo shop. - Stages: monitor (probe), plan (incident), create (Duo), verify (proof), secure (scanners), package (image in both registries; releases link a package manifest and CycloneDX SBOM, e.g. v0.1.99), govern (risk gate, CODEOWNERS), release (staging, promote, GitLab release), configure (
deploy/service.json, checked by a read-onlyconfigure:driftjob after each release). - Deployment: Google Cloud Run in europe-west9, keyless via Workload Identity Federation. Staging is a tagged no-traffic revision; rollback is traffic back to the previous revision.
- Built solo with Claude Code.
Challenges and lessons
- Duo flows start from user actions: the pipeline assigns the incident to the Duo service account, the gate requests the Approval Brief's service account as reviewer.
- Probes are split into
smoke(release) andregression(schedule); running all of them at release would have stopped the bad change before production. - A passing test proves nothing if the agent edited it, and neither does a gate the merge request can change.
Limits
GitLab does not enforce the human approval (that needs Maintainer settings not set here). Timeline events and the feature-flag kill switch came after the first run. The endpoint-to-file mapping is a fixed table, not a diagnosis.
