ProofPack

Inspiration

A merge request can pass review and still not be proven ready. Reviewers end up digging through pipeline pages, test reports and logs to work out what actually passed, what failed and what is simply missing. AI makes this worse: it can propose a fix in seconds, but a confident explanation is not evidence that the fix works. We wanted one place where the evidence is explicit, and where the decision does not depend on anyone's opinion, including an AI's.

What it does

ProofPack builds a structured readiness dossier for a GitLab merge request from real CI evidence (lint, tests, package build, configuration validation and an optional dependency audit). A deterministic policy turns that evidence into exactly one decision:

• blocked: a required check failed
• incomplete: a required check is missing, unknown, not run, or "passed" without an evidence pointer
• ready_for_human_review: every required check passed and has evidence

A missing, empty, malformed or hostile report is never a pass. In a real local run we deleted the unit-test report: four checks still passed, but ProofPack returned incomplete, named what was missing, and the gate exited non-zero. No AI output is an input to the decision, and the gate recomputes the decision from the checks instead of trusting a stored value. The dossier is JSON, Markdown and HTML, with links to the pipeline and job. It states its limits: it covers configured checks only and does not certify code as bug-free or approved for release.

Duo recommends. CI verifies. ProofPack organizes the evidence. A human decides. A blocking, manual release-approval job keeps the release decision with a person.

How we built it

• Python 3.10+, pytest and Ruff, with no runtime dependencies.
• policy.py holds the decision logic and policy validation. evidence.py has the parsers (JUnit XML, Ruff JSON, pip-audit JSON, wheel and sdist verification), secret redaction and report validation against a versioned JSON Schema. report.py renders Markdown and HTML, and cli.py provides validate-config, build-evidence, render and gate.
• GitLab CI/CD: independent check jobs that upload artifacts even on failure, an evidence-report job that runs even after failures, a readiness-gate that fails unless the policy says ready, and the manual release-approval job.
• GitLab Duo Agent Platform, Code Review Flow. We requested @GitLabDuo as reviewer on a deliberately defective merge request (one line changed half-up rounding into truncation). Duo left an inline comment on discounts.py:21 citing our own review instructions, with a concrete example (999 cents at 15% should cost 849, not 850) and a suggested line. It stated that it did not run any tests. We applied exactly that line in a separate merge request, and CI verified it: 141 of 141 tests passed and the dossier read ready_for_human_review. The defective merge request's own pipeline stays red (139 passed, 2 failed, blocked).

Challenges we ran into

• Our first Duo session failed. Its workload died with No module named pip. We pulled the exact workload image and found it is RHEL 9 with Python 3.9 and no pip, while our project needs Python 3.10 or newer, so fixing pip alone could never have worked. We also realised the agent setup script runs outside the sandbox with credentials in its environment, on the merge request's own commit, so installing the repository there would execute code the MR author controls. The fix installs Python 3.12 and only pinned, wheel-only tools, never builds repository code, and keeps the default sandboxed image. Guard tests enforce this. The next review request completed.
• Fail-closed evidence. Turning many different CI outputs into one model without ever letting absence look like success meant treating every unreadable report as unknown.
• Suggestion is not evidence. We had to keep the AI's recommendation strictly separate from what executable checks establish.

Accomplishments that we're proud of

• A genuine, completed Duo Code Review Flow run whose finding we verified against the code and the tests.
• A fail-closed demonstration anyone can reproduce from a fresh clone, plus 141 passing tests covering the decision matrix, malformed evidence, redaction and an end-to-end defect, blocked, fix, ready run.
• Being honest about limits: the demo defect lives only on a demo branch, and neither demo merge request is merged.

What we learned

Agents are most useful for investigation and proposing repairs when verification stays with reproducible checks. Automation that runs before a sandbox needs the same scrutiny as production code. And the most convincing thing a readiness tool can show is how it fails.

What's next for ProofPack

More evidence adapters, per-job evidence links in the dossier, a "pipelines must succeed" merge check, and measured review-cycle data. We have not measured any review-time saving, and no external users have tested ProofPack yet.

Built With

• gitlab
• gitlab-ci-cd
• gitlab-duo-agent-platform
• html
• json-schema
• pip-audit
• pytest
• python
• ruff
• setuptools