Patchwide

A security fix isn’t shipped until every user is safe: GitLab Duo flows that carry one fix across every release line and into every consumer, with proof.

Comment

Inspiration

The fix for a vulnerability is usually written quickly. Then most users never get it. In the week we measured, 23,861,380 of path-to-regexp's 236,172,778 weekly downloads (10.10%, docs/benchmark/npm-vuln-share-2026-10-16.txt) were still versions vulnerable to CVE-2024-45296, two years after upstream fixed every line. Across ecosystems, about 95% of vulnerable downloads already had a fixed version (Sonatype). Fixes come bundled with unrelated changes (up to 85.72% of commits, arXiv:1907.03407), back-ports across branches are the largest multi-patch category (830 of 1,646, arXiv:2607.13206), and nobody owns the question "is everyone safe yet?".

What it does

Patchwide runs a campaign from one confidential advisory to every consumer:

  1. A maintainer assigns the advisory to the campaign-plan flow (Assign trigger). It runs the repro on every supported line: each line is affected, or not affected with a proof.
  2. The line-patch flow writes a security-only patch per affected line, in a private security project.
  3. Our CI/CD component proves each line: repro red before and green after, the full suite green, an empty public-API diff, and a ReDoS class check. A failing line becomes "needs human" and never blocks the others.
  4. The maintainer merges each proven line: the Supervised gate. CI, not an agent, then publishes, signs (cosign keyless), writes an SBOM, OSV and OpenVEX, and creates a GitLab Release.
  5. The release pipeline's Pipeline-passed trigger starts the propagate flow, which opens an upgrade MR in every consumer. A failing consumer pipeline starts regression triage.
  6. The advisory closes only when every in-scope consumer is merged and, where it deploys, deployed and healthy (a ReDoS latency canary on Cloud Run). The report records time-to-safe.

On a seeded replay of CVE-2024-45296 across 5 lines and 3 consumers, advisory to last consumer safe took 4454 s (1 h 14 min, run 87712127-12), including the owner's merges, for 36.225 credits (docs/benchmark/README.md, an upper bound).

How we built it

Agents decide, pipelines execute, the control plane keeps score. Four GitLab Duo custom flows do the judgement work. A CI/CD component runs everything deterministic or irreversible. One Next.js 16 service on Cloud Run receives GitLab webhooks into an idempotent Firestore inbox, processes them through Cloud Tasks, holds the campaign state machine (versioned zod events), renders the tracker on the advisory, starts follow-up flows through the Flows API and serves public campaign pages. Infrastructure is OpenTofu with GitLab-managed state, applied from CI through Workload Identity Federation. Details: ARCHITECTURE.md.

Challenges

  • The Supervised gate could not be an MR approval: GitLab refuses an approval by the account that started the flow session, so the maintainer's merge is the gate.
  • Flows never trigger flows, and flow callbacks are unavailable in our namespace, so the control plane polls sessions and starts what no human action covers.
  • An embargoed fix must not leak: patches live in a private security project, public branches are neutral, and the disclosure is published only after release.
  • Honest proofs: our released patches pass every proof, yet fail some of upstream's own fix-commit tests on 1.x, 3.x and 6.x. We report that (docs/limitations.md).

Accomplishments

  • Real releases on real pre-fix code of a real library, with proof bundles, signatures, SBOMs, OSV and OpenVEX, and consumer upgrades merged and deployed.
  • A benchmark with failures reported: 3 runs, 7 line rows, including a seeded issue no model can have seen (docs/benchmark/README.md).
  • The flows are public in the AI Catalog and the component installs by path: "Enable in 3 steps" took 1m 08s on a fresh import, and the plan note arrived 3 min 47 s after the assign (docs/enable-run.md, session 9902916).

What we learned

Delivery, not the fix, is the bottleneck, and a fix earns trust when it carries its proof. Most of the engineering was idempotency and embargo discipline, not prompts.

What's next

Self-managed GitLab, a multi-tenant control plane, Maven/Cargo/Go adapters, a GitHub App bridge for consumers outside GitLab, embargo coordination with the CVE request, and an "upstream tests" proof (docs/roadmap.md).

Built With

  • api-extractor
  • artifact-registry
  • cloud-monitoring
  • cloud-scheduler
  • cloud-tasks
  • cosign
  • cyclonedx
  • firestore
  • gitlab-ci/cd
  • gitlab-duo-agent-platform
  • google-cloud-run
  • next.js
  • node.js
  • opentelemetry
  • opentofu
  • openvex
  • osv
  • playwright
  • react
  • recheck
  • secret-manager
  • typescript
  • vitest
  • workload-identity-federation
  • zod
Share this project:

Updates

Submission history