Verified Recovery

Inspiration

On a small open-source project, the person fixing a bug often has to handle everything around it too. They reproduce the report, write a test, fix the code, check the scans and make sure the deploy worked.

We wanted to give GitLab Duo an issue and have it reproduce the bug and propose a fix. We kept the test separate from the fix so the maintainer could inspect the record before authorizing a release.

What it does

We added Verified Recovery to an existing MIT-licensed Markdown editor. The recovery workflow starts with an incident issue. Duo's reproduce flow reads the source and returns a regression test, which the operator runs against the incident commit. The test has to fail on an assertion before the operator commits it and pins its hash.

The operator gives the pinned test to a fresh Duo session to propose a source fix. The controller applies only the requested source file, then the operator commits the change and opens a Draft merge request. CI runs the identical test on both commits and requires it to fail before the fix and pass after it. A role check rejects edits outside each role's allowed files. SAST, Secret Detection and Dependency Scanning must all finish, and a High or Critical finding blocks recovery.

CI builds the release bundle, then waits at the blocking approve-recovery job for owner authorization of one exact commit. Once authorized, CI builds the container image and deploys it to Cloud Run by image digest. It checks the live version and every served file hash. Before merging, the controller checks the commit again.

Incident 4 involved an uppercase task marker bug. The injected source had a comment naming the bug's location and cause, and Duo quoted that comment in its saved proposal. Duo therefore had a visible hint for this test and repair; it did not locate the bug unaided. It wrote a test for - [X] and a one-line parser fix. The first verified fix arrived about six minutes after the issue. Deployment took another attempt: Cloud Run rejected the CPU and memory configuration on the first approved deploy. The smoke job stayed skipped, and the merge stopped. We set cpuIdle: true, rebased and ran the checks again. The new candidate passed approval, deployment and smoke verification before MR !6 merged.

The owner made each approval decision. For incident 4, they instructed their browser agent to run the approval job. The record in approval-provenance.json says who clicked and how. Incidents 2 and 3 used owner-authorized API calls.

All four recoveries, VR-1 through VR-4, now show as passing in the live ledger. Each runs the same pinned test before and after the fix. Three incidents used separate test and fix sessions; the four incidents used eight Duo sessions in total. The time from issue to verified fix ranged from about six to fourteen minutes. No High or Critical findings were merged.

Recovery	Injected regression	Result	Issue to verified fix	Merge request
VR-1	Newline word count	PASS	8m48s	!2
VR-2	Bare-CR Markdown preview	PASS	14m11s	!4
VR-3	Lossless Markdown export	PASS	6m21s	!5
VR-4	Uppercase task markers	PASS	5m55s	!6

We deliberately injected all four regressions. The timings end at the first passing recovery gate, before approval, deployment and merge. We have not measured a manual baseline.

The live service runs main commit 6c2fdb8; /version.json lists the commit and hashes of all 12 served files. Release pipeline 2992860523 deploys this commit through approve-recovery, deploy-editor and smoke-editor. The entrant authorized this release; the controller ran its approval job through the GitLab API.

approve-recovery is a blocking manual job. It is not protected: our Developer account gets HTTP 403 when configuring protected environments. A user who can remove Draft or replace root CI can bypass the controller. These checks bind the workflow to an approved commit, but project-wide permissions remain outside what they enforce.

How we built it

We started by importing mooceanstudio/markdown-editor, keeping its MIT license. During the hackathon, we added two GitLab Duo Agent Platform flows along with the controller, CI verification, recovery records, the ledger and Cloud Run deployment.

Both the reproduce and fix flows use read tools and return JSON proposals. Before applying a file, the controller validates the proposed path. We load the verifier and incident pins from a pinned policy commit so a candidate cannot rewrite the rules used to accept it.

GitLab CI runs the role check, regression verification and security scans, with release approval handled by the blocking manual job approve-recovery. Deployment uses GitLab identity tokens and Google Cloud Workload Identity Federation. It pushes the container to Artifact Registry and deploys its digest to Cloud Run, where the smoke check compares the live service with the build manifest.

The public ledger reads committed recovery records containing Duo's final answers, test hashes, TAP output, scan results and approval provenance. The same Cloud Run release serves the editor, ledger and version manifest.

Challenges we ran into

The first deploy for incident 4 failed after approval, stopping the pipeline before smoke verification and merge. We had to fix the Cloud Run CPU allocation and submit a new candidate for another approval. Duo's parser change stayed the same.

Duo's first test proposal for incident 2 hung. The operator rejected it, and another session produced a test that failed on the regression. Our own controller change in MR !3 was also blocked by a High SAST finding. We cleared it by restricting API requests to fixed hosts, without suppressing the rule. There are separate limits to the scan coverage: the evidence verifier and merge helper each retain an inline regex_dos suppression, and Dependency Scanning covers a lockfile with no external npm dependencies.

Accomplishments that we're proud of

We have identical-test red/green proof for all four recoveries. The records also include what went wrong: a rejected proposal, a security block and a failed deploy, followed by successful recovery. Public links let judges follow each issue through the patch and pipeline to the live result.

What we learned

We learned to tie each approval to one commit. In incident 4, the parser fix stayed the same, but changing the deployment configuration produced a new candidate that needed another approval.

For a failing test to be useful, we need its specific cause and exact source commit. Pinning the test's bytes before asking for a fix gives CI a stable target. Keeping approval, deployment and smoke verification separate also helped us diagnose the failed deploy. We could see where the pipeline stopped, and the new candidate required approval.

What's next

We plan to connect the controller stages in one scheduled workflow that pauses for release approval. We also want to add staging, promote the same container digest once its smoke checks pass, and test more editor behavior by extending the incident set.

Built by ArcSolve.

Built With

• ai-catalog
• artifact-registry
• custom-flows
• dependency-scanning
• docker
• flows-api
• gitlab-ci/cd
• gitlab-duo-agent-platform
• google-cloud-run
• javascript
• nginx
• node.js
• python
• sast
• secret-detection
• workload-identity-federation