Release Proof

Inspiration

A green source-test suite can still ship a broken package. A developer's environment may already contain an undeclared dependency, or a resource may exist in the checkout but never make it into the wheel. Release Proof asks a practical question for Python maintainers: does the exact installed artifact actually do what its README promises?

What it does

Release Proof combines GitLab Duo Agent Platform with a frozen installed-consumer acceptance contract. GitLab's existing Developer flow reads a failure issue, diagnoses the packaging defect, changes the permitted metadata, opens a draft merge request and lets GitLab CI test the exact repair head. A person reviews the resulting diff and evidence before any merge.

We demonstrated two real red-to-green repairs on original, MIT-licensed sample packages:

• SessionLedger: source tests and pip check pass, but its installed command fails because a runtime dependency is undeclared
• GreetCard: source tests and pip check pass, but its installed command fails because a required text template was omitted from the wheel

For both cases, fresh Python 3.12 and 3.13 consumer jobs receive artifacts without a source checkout, install locally from wheel artifacts into new virtual environments, and execute the documented command. Acceptance checks the expected output file and stdout. A broken candidate exits nonzero; it is not converted into a green expected-failure result.

How we built it

We built the original fixtures, Python verifier and GitLab CI configuration during this hackathon. The checker and original tests are pinned to an immutable setup commit and SHA256 digests. The second case adds a declarative consumer contract and verifies the candidate CI bytes against the frozen template. It generates JSON, JUnit and a readable acceptance summary containing commit, job, pipeline, checker, contract and artifact hashes.

Duo was invoked through public project issues. It authored each minimal packaging-metadata repair and opened a draft MR. We then checked the entire diff against the corresponding red baseline and inspected both exact-head consumer logs. The second generic checker also passed local controls for both packages without changing checker code, using different contract data.

Challenges and what we learned

Separating an expected-failure demonstration from genuine candidate acceptance mattered. A green badge alone is too weak: source tests can miss release defects, and a changed pipeline could omit its own checker. Frozen behavior plus independent diff review provides a more defensible boundary. We also had to preserve offline pip bootstrap artifacts, distinguish infrastructure errors from package failures, and retain the evidence beyond default artifact expiry.

Verified results

The public repository README links the complete evidence chain:

• Dependency repair: draft MR !1; red pipeline 2922902498; passing repair pipeline 2922927951
• Resource repair: draft MR !2; red pipeline 2923060067; passing repair pipeline 2923096871

Both repair pipelines passed on Python 3.12 and 3.13. Independent review confirmed candidate.toml was the sole changed file and the allowed metadata field was the only semantic change. All 15 relevant build and consumer jobs were explicitly marked Keep. Both MRs remain draft and unmerged.

Scope and next steps

This is Start Fresh / Supervised automation for planning, code repair, verification and packaging. It does not claim production deployment, a Google Cloud bonus, hostile-code isolation, an unseen benchmark or a general repair-success rate. The fixtures and expected behavior are visible. Hosted jobs retain network access; offline installation and fresh venvs are not a security sandbox. Historical Stage 2 hosted CI used its earlier checker; unchanged generic-checker reuse across both packages was verified locally.

The next useful step is testing the contract with consenting maintainers on real packages and measuring whether it catches release defects that their existing checks miss. RevLab's contribution is reviewable installed-artifact evidence, with human approval retained at the end.

Built With

• gitlab
• python