Undo Drill

Inspiration

Teams rehearse deploying forward, rarely backward. A release that changes how data is stored can leave the previous release unable to read what the new one wrote. Each version passes its own tests, so nothing fails until someone rolls back and the rollback makes the incident worse: a one-way door. Small teams with no SRE find out during the incident. Rollback safety is established practice (see AWS: Ensuring rollback safety during deployments); Undo Drill automates the rehearsal and the repair inside GitLab.

What it does

1. A CI job, compat-drill, lets the new release write and the previous release read it back, then reports REVERSIBLE, ONE_WAY_DOOR, INCONCLUSIVE or NO_BASELINE as JSON and an exit code. A one-way door fails the pipeline.
2. A person mentions a GitLab Duo custom flow on the merge request. The flow reads the drill's job log, classifies the door, opens a fix merge request that makes the change expand/contract compatible, and posts a release dossier on the MR and a note on the linked issue. This ran twice on the submitted project.
3. After merge, the pipeline deploys a Cloud Run candidate with no traffic, switches traffic to it, back to the previous revision and forward again, and times the rollback (earlier project).
4. Production is one blocking manual job, promote-production. A person pressed it once.

Evidence (all public, no sign-in needed)

1. MR !4 is red: job 17471199380 says ONE_WAY_DOOR.
2. A person mentions the flow on !4. Session 10072192 ran 119.5 s and opened fix MR !5.
3. !5 is green: job 17477379867, REVERSIBLE, 7 of 7 steps.
4. A person merges !5 into the feature branch v2-rerun (merge commit 75f2df1e; not into main).
5. !4 reruns and turns green: job 17479886109, REVERSIBLE, 7 of 7 steps.
6. Production (earlier project, jg971402/undo-drill) runs the plain-JSON release: /version shows 46e31401.

Full evidence table with pipeline ids and raw-log hashes: docs/SUBMISSION.md and docs/EVIDENCE.md in the repository.

Three limits: (a) both runs are the same JSON-to-gzip change; two runs are not a benchmark. (b) The cloud half ran in another project and deployed 46e31401; the fixed version was never deployed. (c) Cloud Run revisions share no data, so the 15.5 s rollback is traffic switching plus a health check, not a data rollback.

How we built it

GitLab Duo Agent Platform custom flow (three AgentComponents: triage, fix_mr, release_dossier, with a conditional router; flows/undo-drill-dossier.yaml), Mention trigger, GitLab CI/CD (build, unit tests, SAST, pip-audit, compat-drill), container registry, keyless Google Cloud access with Workload Identity Federation, Cloud Run, Artifact Registry. Python, Flask, gunicorn, pytest.

Challenges we ran into

• Cloud Run revisions have no shared disk, so the cloud drill can only prove traffic switching and health; the data guarantee has to come from compat-drill.
• "The previous release" defaults to main; DRILL_DEPLOYED_REF points the drill at what actually runs in production.
• Workload Identity Federation is set up only in the earlier project, so the Google Cloud jobs are skipped on Showcase and a gcp-skipped-notice job says so.

Accomplishments that we're proud of

The loop closed on public data: red drill → human mention → agent fix MR → green drill → human merge → original MR green, with every step linkable.

What we learned

An agent is most useful when a deterministic check gives it evidence to read; the human approves the outcome, not the steps.

What's next for Undo Drill

A shared Cloud Storage volume for staging so the cloud drill can test data, other door types (schema migrations, renamed environment variables), and a production traffic-shift drill.

Honest limits

• Two flow runs (n = 2), same change; no success rate is claimed.
• The video shot at about 0:12 shows "Ready to merge" next to a failed pipeline because "Pipelines must succeed" was off at recording time; it was switched on 2026-10-27 and MR !2 now shows "Merge blocked".
• "The old version can read everything the new version writes" in the video means the drill's seed through notes_app.cli, not all possible data.
• The session page is not public; the evidence is pipelines, job logs, MRs and notes.
• Threat model: accidental compatibility breaks by a trusted team, not untrusted merge requests.

Built With

• artifact-registry
• flask
• gitlab
• gitlab-ci
• gitlab-duo
• gitlab-duo-agent-platform
• gitlab-sast
• google-cloud
• google-cloud-run
• gunicorn
• pip-audit
• pytest
• python
• workload-identity-federation