Backstop

A release loop for teams with no second reviewer: it checks every deploy, opens the incident at the line, proves the fix on a preview copy of production, and closes the incident when the fix ships.

Comment

See it run: incident #3 → fix MR !13 → verify:production red on the deploy of !12, then verify:production green replaying the incident's request after the fix → incident closed with the audit record and the label loop::fix::verified. Run 2, same day: MR !14 re-introduced the defect on purpose; verify:production caught it with the replay set (FAIL replay: incident #3); incident #4 → fix MR !16 → preview:verify: LOOP_REPLAY ... "base": 500, "fix": 200, "verdict": "PASS". Both runs on 27 Oct 2026; the project history is unedited.

Inspiration

You ship alone: no second reviewer, no staging, nobody on call. Tests pass, the preview looks fine, the deploy goes out, and the first sign of trouble is a client saying the page is broken. Then the whole evening goes into finding the line, writing the fix, and hoping it is really fixed this time.

We wanted the part of the team that watches the release: not another reviewer before the merge, but the colleague who notices production is broken, finds the line, proves the fix on real data, and only calls it done when the fix is live.

Two AI Engineer 2026 talks shaped the framing: Matt Pocock's "Fixing the PR Bottleneck" (green CI can lie; never write the same review comment twice) and Gautam Korlam's "Rebuilding CI for AI-Scale Commit Volume" (a pipeline status is state; an audit reads records). Both describe brakes before merge. Backstop is the brake after deploy: production's own failing request is a check an agent can't fake, every incident's request joins a permanent replay set, and the close is an audit record.

What it does

Backstop is a loop around one change:

  1. Verify the deploy. Right after the production deploy, a pipeline job runs the smoke checks and every request the loop has ever recorded against the live URL. If anything returns a 5xx, it reads the app's error record from the production logs and prints one structured failure line with the method, route, exception, source frame and trace id.
  2. Open the incident at the line. The failed pipeline fires a GitLab Duo flow. The flow reads the job log, opens a GitLab incident (not a plain issue) that names the responsible line, asks GitLab Orbit for the callers of the failing function to work out which routes are affected, and records the failing request as a replay record: method, route, status, frame and commit, never request contents.
  3. Draft the fix. The same flow writes a fix on a branch, adds the replay record to the repo and opens a draft merge request linked to the incident.
  4. Prove it on a preview copy. The fix MR's pipeline builds a preview on the same VM as production: one PostgreSQL with two databases restored from a masked dump of production, the MR's migrations applied to one of them, and both the current and the fixed app image running side by side. The recorded request is replayed against both. The verdict is PASS only if the current code fails and the fix passes; the affected routes are checked on the fix copy too.
  5. One human approval. The builder reads the incident, the MR and the replay verdict, and approves. That approval is the only human step, and it is the gate before production.
  6. Close on the verified deploy. The merge deploys; the post-deploy check runs again and passes; a second flow writes the audit record on the incident (what failed, what was replayed, what the gate said, who approved, which commit shipped) and closes it. GitLab measures time to restore from incident open to close, so closing on the verified deploy, not on the merge, makes that number mean "fixed in production".

What is new is the gate: the exact request that broke production must fail on the current code and pass on the fix, on a masked copy of production's data, before a human can approve. The incident closes only when the verified fix is live. The whole loop is visible in GitLab: the incident, the MR, the preview environment, the pipeline history and the DORA dashboard.

What previews catch and what production catches. The preview copy catches what the code breaks. The post-deploy check plus the replay catches what production's data breaks: the class of bug tests miss because test data never has it. The demo bug is exactly that kind. A new "Last booked today / yesterday" line on the timetable passes every test and the preview, because every test booking is seconds old. Production has a booking from days ago, so the home page is a 500 the moment the change deploys. The loop takes it from there.

The example app. Third Floor Studio is a small class-booking site: a week's timetable with mats drawn as spots, a class page with a plain HTML form, a signed cookie that remembers your mat, and a confirmation page. It exists to give the loop something real to watch. Visitors are asked to use made-up details. Live at http://8.234.102.148.

How we built it

Where this differs from GitLab's own flows: Fix CI/CD Pipeline repairs a pipeline that fails; this loop handles a change whose pipeline is green and that still breaks production. GitLab already turns alerts into incidents; this loop adds the line, the blast radius, a replayed proof and a verified close.

  • Two custom Duo Agent Platform flows, registered in the AI Catalog and run by pipeline triggers: "Loop: incident and fix" on a failed push pipeline on main, "Loop: verified close" on a successful one. Each has a triage agent and a worker agent with a narrow toolset. The flows query GitLab Orbit for the blast radius of the failing function. Definitions in .gitlab/duo/flows/.
  • GitLab Duo Code Review on every MR, including the loop's own MRs. Four review rounds on the kernel MR ended in Duo's own approval. Review is advice, and on a team of one it gets overridden under deadline. The demo does exactly that: Duo flagged the planted bug, the builder merged with a documented override, and the loop is what catches it in production.
  • CI/CD: tests on the production base image, SAST, secret detection, dependency scanning and container scanning (164 findings on the Debian base image, 0 after moving to Alpine), an image build pushed to the project registry and pulled by the deploy, keyless deploy to Google Compute Engine through OIDC and workload identity federation over an IAP tunnel, verify:production after every deploy, preview:verify and preview:stop for every MR with a GitLab environment and 24-hour auto-stop.
  • Gate integrity: the preview and replay harness is always taken from main, never from the MR under test, and any MR that touches the gate is labelled so a reviewer sees it.
  • Records: replay records in loop/replays/, structured LOOP_ERROR, LOOP_FAILURE and BLAST_RADIUS lines in logs, the incident and audit comment in GitLab, labels loop::fix::verified. Everything is queryable from Orbit and from the GitLab API.
  • Config: one file, .gitlab/loop.yml, holds everything project-specific: the production URL and verify job, the mask list for preview data, the record allowlist (method, route, status, parameter names, never values).
  • One image, built once. The image is built once per commit, tagged by SHA, tested on the same Alpine base as production, scanned, and that exact tag is deployed; needs: lets scanning and previews start as soon as the build is done, and the uv cache keeps the test job short.
  • The app: server-rendered FastAPI with PostgreSQL, SQL migrations run by dbmate, OpenTelemetry on every request, one Docker Compose stack on an e2-small VM.

Lifecycle stages, with what each one is in this project:

  • Create: merge requests with GitLab Duo Code Review on every one, including the loop's own.
  • Verify: the test job, preview:verify (the replay gate on the fix MR), verify:production after every deploy.
  • Package: one image per commit, built and pushed to the project registry, the same tag deployed.
  • Secure: SAST, secret detection, dependency scanning and container scanning on every pipeline.
  • Release: the production environment, deployments, the MR approval as the only gate.
  • Configure: Compute Engine deploy code in deploy/ (keyless OIDC to workload identity over IAP), preview environments with 24-hour auto-stop.
  • Monitor: the post-deploy check, the OpenTelemetry error record, GitLab incidents, the DORA dashboard.
  • Govern: the approval rule as the one human gate, the audit record on each incident, gate files labelled when a change touches them.

Challenges we ran into

Working out what the Duo flow runtime can and cannot do took a week of probes, kept as an open draft MR in the repo: MCP servers are excluded in flows, but Orbit answers through the runtime's pre-authenticated glab; flow schedules need a service account a Developer cannot create, so pipeline triggers carry the loop; the one human approval is an MR approval rule, because protected environments need Maintainer.

The preview harness went through four Duo review rounds before it was trusted to sit in the gate: masking that could silently fail, a teardown that leaked previews after merge, port collisions between previews, unvalidated identifiers in generated SQL. Each round is in the MR history.

Finding a demo bug that tests and previews genuinely miss took real work. The first candidate failed the new UI's tests; the final one depends only on how old production's data is.

Accomplishments that we're proud of

  • The loop ran end to end on the live project on submission day, twice: incident, fix MR, approval, verified deploy and replay, closed incident. The second run shows the gate itself: the same request at 500 on the current image and 200 on the fix, on a masked copy of production, before the human approved. All of it is unedited in the project history.
  • On the first run, the close flow refused to close the incident because it could not find the preview verdict on the fix MR. It closed only after a human confirmed the approver, and its audit record says the replay was verified on production, not on a preview. A gate that can say no is the whole point.
  • A gate that cannot be fooled by the change it is testing: the harness comes from main, the data is a masked copy of production, and the verdict needs the current code to fail.
  • No long-lived keys in the repo or CI: keyless deploys through workload identity federation, masked preview data, replay records that never hold request contents.

What we learned

  • Previews catch what the code breaks; only production's data shows what the data breaks. A release loop needs both halves.
  • Treat the platform as something to measure, not to assume. The platform decisions here each follow a probe that is still in the repo.
  • Close incidents on the verified deploy, not on the merge, or your time-to-restore number is a lie you tell yourself.
  • The Door and Blast Radius ideas come from Matt Pocock's pr skill (mattpocock/skills, MIT); here they run as a gate check and an Orbit query.

What's next for Backstop

First, a small ordering fix in the incident flow: it pushes the fix branch a few seconds before it opens the merge request, so the fix MR's first pipeline can lack the replay job (on the day, a pipeline started after the MR existed ran it). Then the migration audit before merge, cut for time: read the SQL, run it on the preview copy, report before the human approves. Then a loop status page in the app, the Orbit query that renders the full audit trail across incidents, GitLab Observability in the audit record, and running the same flows unchanged on a second small app from a different domain.

Supported scope, honestly: one database family (PostgreSQL), one containerised app behind one URL, OpenTelemetry for the error record, single-request failures. DORA figures come from this project only. GitLab's dashboard for 27 Oct, the day of the two runs: 5 production deployments, change failure rate 40%, median time to restore about 19 minutes from incident open to verified close. Six data points, not a trend. The DORA dashboard needs GitLab Ultimate; the loop itself (flows, incidents, preview replay, verified close) does not.

Built With

Share this project:

Updates

Submission history