Rollback Lab

GitLab Duo repairs cross-version failures. Independent checks prove accepted orders survive before the tested image is promoted.

Comment

Inspiration

Your code went back. Your data didn't.

A newer release can accept customer orders that an older worker cannot read. Rolling the worker back may restore a familiar process while leaving those accepted orders stranded. The useful question is not just whether the service starts: did the promises made to customers survive?

Rollback Lab is an evidence-driven recovery workflow for engineers responsible for APIs, asynchronous workers and durable state. It asks GitLab Duo to repair a measured compatibility failure, then gives an independent evaluator the authority to accept or reject the candidate.

What it does

The central experiment reproduces a new-message/old-worker mismatch. Thirty orders have been acknowledged, but the old worker cannot finish them. A native GitLab Duo flow proposes application changes and opens a merge request. A fixed broker admits changes only to three designated application source files; the workload, contract, fixtures, receiver, verifier and CI/build configuration stay outside the candidate's repair scope.

Protected GitLab CI evaluates the candidate against three sealed fixtures. A separate release exercise pulls the exact tested image and checks recovery on the same live database used inside that exercise. It does not restore that database or rebuild the evaluated image. Independent observations check accepted work, amounts, tenant boundaries, duplicate effects, receipt records, resumed intake and retained history. Monitoring and cleanup have their own evidence.

In the new recorded native run, 30 accepted orders were retained, ten new orders completed, and 40 orders matched 40 confirmation receipts, with zero logical duplicates and zero unfinished work. All 15 release business checks passed. These are synthetic workloads on real PostgreSQL and actual HTTP/worker/receiver execution, not customer-production results.

The website makes those checks inspectable. Download an evidence bundle and import it into Verify evidence: the browser recomputes the business assertions locally. A deliberately altered copy with a missing receipt is rejected even after its data hash is recomputed and its original success claim is left intact.

How we built it

GitLab Duo Agent Platform supplies the native recovery flow. Its configured roles cover change planning, experiment planning, repair engineering and reporting. Actual native actions and committed source are reconciled separately for each run; not every configured role is assumed to have written output in every invocation.

Python implements the application, durable control ledger, proposal broker, independent evaluator and portable verifier. PostgreSQL stores application and control state with separate privileges. Docker isolates the protected evaluation and records the actual tested artifact identities. GitLab CI/CD handles evaluation, packaging, release observations, monitoring, cleanup and signed evidence publication. A React/TypeScript/Vite interface is hosted on Vercel and GitLab Pages.

The main source intentionally retains the faulty application as an experimental control. Candidate repairs live in separate native merge requests; judges can replay original native patches using the documented Docker recipe. This replay creates new workload observations but does not make a new model call. Fresh native generation remains a distinct, authorized GitLab Duo workflow.

Challenges we ran into

The difficult boundary is preventing an apparent repair from rewriting its own exam. We separate repairable source from the evaluator, restrict admission before execution, and verify raw observations rather than trusting a green process or a reported PASS.

We also had to distinguish a successful release job from a completely reported native workflow. One practice invocation reached a passing synthetic release but ended in HOLD after a pipeline-status read timed out. That history remains intact. Bounded retries were added for transient reads without repeating writes; the subsequent filmed native invocation completed with a verified broker report.

Accomplishments we're proud of

  • A real native Duo-generated candidate, MR and protected evaluation/release chain, with source and artifact identities.
  • Fifteen recomputable business assertions, including preservation of accepted work and completion of new intake.
  • Same-live-database recovery within the separate release exercise, using the evaluated image without restore or rebuild.
  • A browser verifier that rejects a semantically inconsistent bundle even when its raw-data hash has been updated.
  • Public source, explicit reproduction instructions, recorded failures and separate evidence for runtime origin and native authorship.

The broader published v2.1 experiment matrix contains 58 protected evaluations, and 30 unsafe semantic mutations were rejected. These counts are tests and replays, not 58 new AI repairs. The documented targeted fault-injection baseline ties our gate on its comparison set; we do not claim superiority over all integration testing.

What we learned

Recovery is about preserving accepted state, not merely returning an image tag to its previous value. Agent output becomes useful when a fixed, independently observed contract can stop an unsafe action. A signature, digest, successful job or confident model report answers a different question from whether customer-visible work survived.

What's next for Rollback Lab

The next research steps are independent third-party reproduction, broader compatibility faults and evaluation of the contract against additional real application designs. The present project is deliberately scoped to controlled synthetic recovery experiments; it does not repair arbitrary repositories or certify general production safety.

Try it and inspect the evidence

  1. Open Rollback Lab and choose See the verified recovery.
  2. Download its recovery evidence, then import it under Verify evidence to recompute the checks locally.
  3. Open Run & reproduce or JUDGE_TESTING.md for the no-account Python/Docker recipe and separate native-generation access requirements.

No-account browser verification and offline bundle verification need no model key. Full runtime reproduction needs Python 3.14+ and Docker with Linux/privileged-container support. Fresh native generation needs GitLab Developer/Duo access and available sponsorship. Hosted reference exercise slots are limited and are not new AI generation.

Video evidence note: The native session, merge request and protected pipeline shown are run 9654831 / MR42 / pipeline 2960694137. The browser upload demonstration uses an earlier practice release bundle from session 9652190, not the bundle from 9654831. That practice release passed independently, although its native reporting workflow ended in HOLD after a read timeout. Run 9654831 later completed with an authoritative verified broker report. The modified upload is a deliberately altered demonstration copy, not an actual failed native release. The website's opening recorded recovery is published session 9635725, also a distinct run. Waiting and navigation are edited for presentation; this is not a continuous real-time execution claim.

Public MIT source: repository. Filmed native repair: MR42. Protected evaluation and release: pipeline 2960694137. Detailed prior-art comparison and limitations: COMPARISON.md.

Built With

Share this project:

Updates

Submission history