-
-
Actual lab proposal: high1 to zero audit findings; entrant accepted reviewed outcome. Working demo separately executes designed checks.
Inspiration
A maintainer reviewing an AI dependency repair needs more than a green pipeline. A proposal can also weaken a scan, replace a test or attach a report from another commit. PatchProof makes the review boundary explicit: did the approved repair happen under unchanged checks?
What it does
PatchProof combines GitLab Duo's real multi-step repair flow with a separately selected committed-snapshot collector and a readable evidence page. The collector reads the exact baseline and candidate as Git blobs, permits only two dependency files to change, checks the baseline-approved version/registry/integrity values, obtains fresh npm audit reports itself and captures unchanged app control tests. CI, tests, policy, collector and unrelated source changes are blocked. The result is EVIDENCE_COLLECTED_REVIEW_REQUIRED or BLOCKED, never permission to merge.
The demonstrated case is a deliberately narrow, metadata-only dependency lab. It does not install or execute lodash or an exploit. It lets a maintainer inspect an actual repair proposal without executing candidate dependency code.
Actual GitLab workflow
Baseline pipeline 2988358049 passed its unit tests but failed a real npm audit with one high-severity vulnerable dependency. GitLab Duo's Fix Pipeline session 10033915 diagnosed the failure, read the policy, edited only app/package.json and app/package-lock.json, ran unchanged checks and opened draft repair MR !3. The flow completed these steps without individual step approvals; the entrant subsequently accepted the reviewed proposal as the outcome. The MR stays an unmerged draft for judge inspection.
The proposal updates lodash 4.17.23 to the baseline-approved 4.18.1. Its hosted pipeline 2988368427 passed 92 tests and an actual zero-finding audit. A separately selected reviewer job 17459557731 then collected new evidence from the immutable commits: eight gates passed, before high 1 / exit 1, after total 0 / exit 0, and four unchanged control tests passed. Its raw reports, TAP, full review, provenance and HTML page are downloadable artifacts. The 27 reviewer-source files were downloaded through the Browser and byte-matched to the selected source.
What PatchProof adds
The standard Duo flow proposes the repair. PatchProof independently asks whether the proposed change respected the review policy and obtains the evidence used to answer that question. A separate, manually designed CI-weakening candidate was blocked before a registry call. This negative case is a functional test, not an observed bad agent proposal.
The preserved offline verifier remains synthetic-only. Its designed 39-case evaluation produced two accepted valid cases and 37 blocked negatives; a deliberately weak tests-only baseline falsely accepted 35 negatives. Those numbers describe designed cases, not real-world accuracy or repair success rates. The live result is one actual proposed repair; no user study or time-saving measurement is claimed.
Claim-to-evidence guide
| Claim | Evidence to inspect | What it establishes |
|---|---|---|
| One actual multi-step Duo repair | Session 10033915, draft MR !3 | Two dependency files changed, followed by human outcome acceptance; one controlled lab proposal. |
| Actual CI changed from audit failure to pass | Baseline 2988358049, candidate 2988368427 | Before high 1; after total 0 and 92 passing tests. No installed-lodash compatibility test. |
| Separately collected fresh evidence | Job 17459557731 and raw artifacts | Eight review gates and four unchanged controls passed for the pinned actual commits. No enforced permission isolation or attestation. |
| Changes to checks are rejected | Functional rehearsal 17459221787, local inspector | A manually designed CI-weakening case blocks; this was not an observed agent error. |
| Designed-case evaluation | Run node scripts/evaluate.mjs |
39 synthetic expected decisions: 2 accepted, 37 blocked; weak tests-only baseline falsely accepts 35 negatives. Not real-world accuracy. |
| Reproducible working interface | Run the offline inspector below | Executes the existing Node/Git synthetic verifier and hashes captured real-job files. Does not rerun Duo or the actual audit. |
The public working-UI video shows this automation executing, including pass, block and byte-check results.
Working project and reproduction
The public source includes a loopback working inspector: run node scripts/evaluate.mjs, then node inspector/serve.mjs and open its printed URL. Its controls actually execute the unchanged Node/Git verifier against designed valid and changed-CI cases. A separate button checks three captured real-job file hashes. It clearly labels synthetic checks and actual recorded evidence. No package installation, registry call, credential, cloud account or card is needed for this offline path. The README also gives exact pinned CLI instructions for a separate fresh registry recollection.
How it was built
Original JavaScript, Node built-ins, Git and static HTML/CSS were authored for this event on October 25–26, 2026. GitLab CI provides the actual hosted jobs; the genuine GitLab Duo Agent Platform Fix Pipeline flow performs the repair proposal. The collector uses fresh temporary npm HOME/cache/configuration and lockfile-only audits with scripts disabled. No runtime package installation, paid service, new subscription, card-backed trial or cloud deployment was used. Source is MIT licensed. Routine concept and implementation decisions were AI assisted under the entrant's delegation; this is not a claim of personal human code authorship or review.
Challenges and lessons
Evidence integrity and execution trust are different. A full commit SHA and matching hashes help detect substitution, but do not authenticate a candidate-controlled job. The separately selected reviewer source and official Node image are inspectable; enforced reviewer permission isolation and cryptographic execution attestation have not been deployed. The source consistency check occurs after code loading.
Git blob framing, UTF-8 BOM preservation, replacement refs, exact lockfile constraints and unchanged test bytes needed explicit handling. Those risks led to concrete parser/Git regression tests and narrower claims. The baseline chooses the maintainer's patched lodash 4.18.1; a reviewed advisory mirror lists 4.18.0, and that discrepancy is disclosed in the repository. Actual npm reports recommended 4.18.1 and had zero findings afterward.
Limits and next steps
The unchanged receipt app is dependency-free. Its four controls do not import lodash and cannot establish dependency runtime compatibility, application exploitability or production safety. Policy support is limited to this exact public direct-dependency update. General transitive updates and production deployment are outside the demonstration.
There is no automatic merge or deployment. The entrant accepted the reviewed two-file proposal; the MR is retained as an unmerged draft for judge inspection. Path A / Supervised is supported by the observed multi-step execution and this human outcome approval; organizers determine final qualification. Next work is enforced independent reviewer execution and broader, properly scoped compatibility evaluation. No competition acceptance, rank, award or guaranteed outcome is claimed.
Built With
- gitlab
- javascript
- node.js
