RecoveryProof X

• 

  The loop: a human states the intent, GitLab Duo plans, writes and reviews, CI rehearses failure and gates, Cloud Run canary ships.

• 

  Live production Passport: 10 required checks PASS, including production_verified and a keyless Sigstore image signature.

• 

  Adopt it with two files: a reusable CI/CD component catches known-bad mutations in any HTTP service. Real CI output.

• 

  RecoveryProof X: every release gets a Recovery Assurance Passport, built from evidence and signed with Sigstore.

Inspiration

Most outages aren't caused by code that fails to deploy. They're caused by code that deploys fine and quietly does the wrong thing. A health check returns 200 while customers are charged the wrong amount. And when teams roll back, they rarely prove the rollback restored correct behavior, or that the root cause was actually fixed. Small teams without a dedicated SRE function have the least time to produce that evidence. We wanted a pipeline that rehearses recovery on every release and hands you proof.

What it does

In 60 seconds

• Problem: a release can pass /health while charging customers the wrong amount, and a rollback is rarely proven to have worked, let alone to have fixed anything.
• What we built: every commit rehearses failure. It detects the regression, rolls back to the last release tag, re-verifies every business invariant and publishes a Recovery Assurance Passport that can say NOT ASSURED.
• Agents: GitLab Duo Planner, Developer, Code Review and Fix CI/CD Pipeline, in 29 linked sessions. A real incident was fixed by Duo and validated by a counterfactual on real commits. One click on an issue reached verified staging in 3 min 53 s. Duo also adopted our reusable component, remediated 8 dependency CVEs and cut the container image from 164 known findings to 0. Duo Code Review caught real flaws in the deploy gate, the signing code and the scheduling, and Duo Developer fixed each of them.
• All the way to Google Cloud Run: canary at 0% traffic, verified, promoted, with automatic rollback armed.
• Adoptable: a reusable CI/CD component runs the same drill on any HTTP service from one recovery.yaml. It is proven on a second service in our own pipeline.
• Measured, not claimed: 207/207 faults detected and recovered across 71 real pipelines, 0 false rollbacks, and 86/86 known-bad mutations caught by the two services that adopted the component.

In detail

RecoveryProof X adds a recovery-assurance layer to the GitLab post-code lifecycle:

1. Rehearses failure on every commit. A recovery lab starts the known-good release and the candidate as isolated processes, built from real git refs. It replays golden orders against an independent pricing oracle over HTTP, detects violations even when /health stays green, rolls back to the last release tag on the same port, and re-verifies health and every business invariant. Timing budgets and a rollback attempt limit come from a versioned release policy.
2. Proves that a rollback is not a fix. For the recorded incident, a counterfactual replays three real commits. The rollback target meets the pricing contract but loses the feature. The defective candidate has the feature but breaks the contract. GitLab Duo's fix meets the contract and keeps the feature. The result records FAILURE_DETECTED → RECOVERY_VERIFIED → INVESTIGATION_LINKED → FIX_VALIDATED with the fix commit and the agent session that produced it.
3. Uses GitLab Duo agents for the parts that need judgment:
   • Planner agent wrote the implementation plan for a new feature (#5): which constant feeds each field, an independent agreement test, and a subtle risk (the tax rate is bound as a default argument). Developer Flow then implemented exactly that plan.
   • Developer Flow fixed the root cause of the incident, implemented features from issues, addressed review feedback and resolved a merge conflict without dropping a single test.
   • Code Review Flow found an edge case nobody pointed it to (a negative slice bound in a fault-injection path), caught a weak test assertion, and approved the fixes.
   • Agents correcting agents (#6 → !24): one click and Duo Developer adopted our reusable recovery-drill component for the production service, with a hand-computed golden contract and two mutations, including incident #2's defect class. Duo Code Review then found that the new drill was not wired into the deploy gates, so a failing drill could not have stopped the Cloud Run canary. Duo Developer fixed it in 2 minutes, and Duo Code Review approved.
   • Agents reviewing security code (!27): Duo Code Review found four real flaws in our keyless-signing change: a broken signer identity for tag pipelines, a check that failed open, a "required" check that was really opt-in, and a shell-quoting injection in the displayed verify command. Duo Developer fixed all four, with tests, in 5 minutes.
   • Security remediation (#7 → !28): one click. Duo Developer added a blocking dependency audit with an SBOM and remediated 8 known advisories in FastAPI/Starlette/pytest with no application change. It reached verified staging in 7 min 56 s.
   • Hardening the image (#8 → !31): Duo Developer added container scanning with a fail-closed gate and moved the runtime to a minimal Chainguard image: 164 known findings (44 High) went to 0. When its own change broke the staging smoke test, it diagnosed and fixed it in the same session. Duo Code Review then caught that the change would have made every hourly production-watch pipeline invalid, and that missing scan evidence passed silently. Duo Developer fixed both.
   • Fix CI/CD Pipeline Flow diagnosed a failing lint job from its log, including a second failure that hadn't run yet, and proposed the exact fix.
4. Gates every deploy on evidence, all the way to Google Cloud. Branch pipelines build a container image, boot it as a CI service, smoke-test it against the pricing contract and publish a staging site. On main, the same image digest goes to Google Cloud Run as a canary: it deploys a new revision with no traffic, verifies it on its own tag URL, promotes it to 100% traffic, re-verifies the live URL, and keeps an automatic rollback to the previous revision ready. Authentication is keyless (Workload Identity Federation trusting only this project's main branch). Both deploy only if every gate passes:
   • the tests and Bandit;
   • GitLab Secret Detection and SAST, behind a blocking security gate;
   • a dependency audit with an SBOM;
   • container scanning of the exact deployed image, behind a fail-closed gate;
   • a keyless Sigstore signature check on the image, which must be signed by this project's own pipeline;
   • the recovery rehearsal, the policy gate and the Passport.
5. Is adoptable by any team, in any language, with two files. A reusable GitLab CI/CD component (recovery-drill) runs the same drill on any HTTP service. The project's recovery.yaml declares:
   • how to start one release;
   • the exact responses the service owes (a golden contract);
   • known-bad mutations. Each mutation must be detected and rolled back, which proves the contract would catch that class of defect.
   • Proven here: our own pipeline consumes the component on a second, unrelated service (examples/shipping-service, standard library only, no FastAPI) and publishes the result under /adoption/. Both mutations (billing weight rounded down, an off-by-one free-shipping threshold) are caught in about 10 ms and rolled back. A defective commit, a mutation the contract misses, or a patch that no longer applies all block the drill.
6. Publishes a Recovery Assurance Passport per commit. On main the Passport is published only after production is verified, and it adds a ninth required check, production_verified. The verdict is ASSURED only when the checks pass: evidence bound to this commit, every fault class detected and recovered, healthy releases not rolled back, business invariants enforced, policy gate, deployed-image smoke test, clean security scans, and the incident fix validated. Otherwise the verdict is NOT ASSURED, with the reasons. A "who did what" ledger links all 29 GitLab Duo sessions, and every agent commit is verified against git (author must be Duo Developer). The Passport itself is signed keylessly with Sigstore by the main pipeline's identity and recorded in the public transparency log. Anyone can run cosign verify-blob; changing a single byte fails it.

Measured results (real pipeline history, not projections)

Aggregated from every recovery-rehearsal artifact in the project's GitLab history (71 pipelines, 354 recorded scenario runs) with python -m recovery_lab metrics. The method is in docs/IMPACT.md.

• 207/207 injected faults detected, rolled back and re-verified (0 missed). 0/69 healthy releases rolled back.
• 86/86 known-bad mutations caught, and 0/43 false rollbacks, in the drills of the two services that adopted our CI/CD component.
• 2/2 defective real candidates rejected, with production restored. 0 slipped through.
• Detection of a pricing regression while /health stays 200: median 18.5 ms. Rollback plus re-verification: median ~0.53 s (worst 2.0 s).
• GitLab Duo Developer fixed the incident's root cause 2 min 14 s after its session started. CI validated the fix 5 min 24 s after the start.
• One click on an issue to a verified staging deploy: 3 min 53 s (feature, #4), 6 min 51 s (Duo adopting the recovery drill for the production service, #6) and 7 min 56 s (Duo remediating 8 dependency advisories, #7). Cloud Run canary from candidate to verified production: 215 s.

These are rehearsals of a small reference service on shared CI runners. They aren't production incident statistics or cost savings, and the human wait before a Duo session starts is excluded.

Autonomy: what we demonstrated, and what we didn't

• Supervised (submitted level). Agents executed multi-step work on their own: plan, fix, test, commit, address review, resolve conflicts, diagnose pipelines. CI cleared every gate and deployed to staging. A human approved the merge, and the canary then promoted the release to Google Cloud Run production on its own, verifying it before and after the traffic switch.
• Intent to verified staging with no human in between. We demonstrated this once: a single click on Implement on issue #4 led to Duo implementing the change and opening !7, then CI ran every gate and deployed to staging in 3 min 53 s (pipeline 2985508546).
• Not demonstrated: Hands-off to production. We tried and documented why it fails on this platform today. /merge from an AI workflow returns 403 Quick actions merge cannot be used with AI workflows. An agent-requested review returned DCR4008. A mention written by one agent doesn't start another agent. Chaining agents natively needs Maintainer-configured triggers, and our role was Developer.

DevSecOps stages covered (all nine)

• Plan: Duo Planner implementation plan for #5; incident issue with root cause and acceptance criteria; intent issues for agent delivery
• Create: Duo Developer commits (root-cause fix, features, review fixes, conflict resolution)
• Verify: 93 tests including negative controls; Duo Code Review; Fix CI/CD Pipeline diagnosis
• Package: Wheel and container image in the GitLab registry, with digest
• Secure: GitLab Secret Detection and SAST behind a blocking gate; Bandit; dependency audit + SBOM (8 CVEs found and remediated by Duo); container scanning of the deployed image behind a fail-closed gate, on a digest-pinned Chainguard runtime with no shell or pip (164 findings → 0, image 36% smaller); keyless Sigstore signatures on the image and the Passport, verified before deploy; fault injection refused in production
• Release: Reusable recovery-drill CI/CD component adopted by a second service; rollback to the last release tag; per-branch staging; Cloud Run canary to production with automatic rollback
• Configure: Versioned release policy (budgets, rollback limits, stop rules); agent config; review instructions
• Monitor: Health plus business invariants, detection and recovery timing, smoke tests of the candidate and live production URLs, and an hourly scheduled production-watch that re-checks live Cloud Run against the pricing contract
• Govern: Commit-bound policy gate, Recovery Assurance Passport, verified provenance ledger, human production approval

How we built it

• Service and lab: Python 3.12 with FastAPI. The lab (recovery_lab/) is a deterministic controller with a state machine, bounded rollback, process-group cleanup and lab-only fault injection (FAULT_MODE is refused outside lab environments, including in the production image).
• GitLab CI/CD: 26 jobs on main (two use our own reusable component, recovery-drill: on the production service and on a second, unrelated service) covering lint, tests (93, including negative controls), Bandit, a pip-audit dependency gate with a CycloneDX SBOM, GitLab Secret Detection and SAST with a blocking gate, keyless Sigstore signing and verification of the image and the Passport, wheel and container packaging to the GitLab registry, the recovery rehearsal, the counterfactual, the policy gate, the Passport, and GitLab Pages for production and per-branch staging (parallel deployments) with GitLab environments.
• Google Cloud: Cloud Run (canary revisions, traffic tags, automatic rollback), Artifact Registry remote repository proxying the GitLab registry, Workload Identity Federation with GitLab ID tokens, and a budget alert. Setup is a script in the repo (deploy/gcp/bootstrap.sh).
• GitLab Duo Agent Platform: the Planner agent plus foundational flows (Developer, Code Review, Fix CI/CD Pipeline), with AGENTS.md, .gitlab/duo/agent-config.yml (so agents can install dependencies and run the test suite) and custom review instructions.
• Honest labelling everywhere: each piece of evidence is tagged deterministic, gitlab_duo or human.

Who did what (disclosure)

• GitLab Duo (agents): implementation planning, root-cause fix, review findings, review fixes, conflict resolution, pipeline diagnosis, feature delivery to staging.
• GitLab CI (deterministic): detection, rollback, gates, packaging, staging deploys, Cloud Run canary, Passport.
• Human (project owner): stated intents, started flows, approved production merges.
• Claude Code (AI coding assistant): wrote the recovery lab, CI configuration, Passport, protocols and documentation.

Where to see the agent evidence without logging in

GitLab Duo session pages (/-/automate/agent-sessions/<id>) require a GitLab login. The same sessions are visible publicly as:

• MR and issue activity lines such as "Duo Developer started session 10009863 triggered by …" (!2, !4, !7, !12, issues #3 and #4),
• the agents' own comments and review threads on those MRs,
• commits authored by Duo Developer (for example 8b6b44c, adbd70c, aa741bc, 83a0613c),
• the Passport's "Who did what" table, which verifies those commits against git.

Challenges we ran into

• Keeping a green pipeline from lying. Health-only checks, stale evidence from another commit, a scanner that silently didn't run, or a "fix" that is really the defect must all produce NOT ASSURED. Each case has a test.
• Platform limits on agent autonomy. We designed three delivery protocols (v1–v3) and documented exactly which hand-offs GitLab allows today.
• A review session that failed on the platform side (no_output). We found that unassigning and re-assigning the reviewer in separate comments restarts the review.
• CI-level details: a shallow clone without git in the slim image broke ref-based releases. The lab recorded it as FAILED rather than hiding it, and we switched images.

Accomplishments that we're proud of

• A real incident, fixed by an agent, and validated by a counterfactual on real commits, not by the agent's own claims.
• Agents that behaved well under constraints. For example, Duo Developer refused to bypass the merge restriction and handed the decision to a human.
• Measured, not claimed: 207/207 faults detected and recovered across 71 real pipelines with 0 false rollbacks, plus 86/86 mutations caught by the two adopting services.
• 29 agent sessions across four Duo capabilities (Planner, Developer, Code Review, Fix CI/CD Pipeline), every one linked; agent commits verified against git.
• A full chain on one feature: Duo Planner plans → Duo Developer implements the plan → CI verifies and deploys to staging → human merge → Cloud Run canary promotes to production → Passport ASSURED with production_verified.

What we learned

Rollback restores the past; only a fix restores the product. Evidence beats dashboards: a verdict that can say NOT ASSURED is worth more than one that only ever says green.

What's next for RecoveryProof X

• Trigger-based chaining (Maintainer) for review → developer hand-offs.
• Automatic rollback from the hourly production-watch (today it verifies and alerts; rollback runs only during a deploy).
• Signing each run's evidence individually (today the Passport and the image are signed; run files are checksummed).
• Publishing the recovery-drill component to the GitLab CI/CD Catalog and adding non-HTTP probes (queues, cron jobs).

Built With

• artifact-registry
• bandit
• chainguard
• claude-code
• cosign
• docker
• fastapi
• gitlab
• gitlab-ci
• gitlab-container-registry
• gitlab-duo-agent-platform
• gitlab-pages
• gitleaks
• google-cloud-run
• pip-audit
• pydantic
• pytest
• python
• ruff
• semgrep
• sigstore
• trivy
• uvicorn
• workload-identity-federation