ShipProof
Ship on proof, not trust.
ShipProof independently checks whether a code change fulfills the promises in its GitLab issue.
GitLab Duo Agent Platform generates acceptance missions, investigates failures, and prepares repairs for human review. GitLab CI/CD enforces the evidence gate and verifies the deployed Cloud Run candidate before promoting production traffic.
Inspiration
Developer tests tell you whether the code passes the tests you wrote. But what if those tests miss the same requirement the implementation missed?
We built ShipProof to independently check whether a change delivers what its GitLab issue promised.
Every acceptance verdict should answer three questions:
- What requirement was checked?
- Which commit was tested?
- Where is the evidence?
Our goal is to help developers and reviewers make release decisions using observable user behavior and traceable execution results.
What it does
ShipProof is a supervised delivery loop that connects issue criteria, independent acceptance evidence, reviewable repairs, and guarded releases.
Our demonstration application is Northstar Store, a storefront on Google Cloud Run where visitors can browse products, manage a cart, and place no-payment demo orders.
The catch: developer tests passed, but a requirement was broken
Issue #7 required that decreasing a cart item from quantity 1 removes it. An empty cart must then show a zero count and disable checkout.
The implementation missed that boundary. The developer-written tests stopped at quantity 1, so they passed.
ShipProof’s independent acceptance mission clicked Decrease at quantity 1 and observed that the item remained in the cart.
| Evidence | Result |
|---|---|
| Developer tests on the original feature commit | Passed |
Independent acceptance at commit 20e6bfcf |
AC5 BROKEN |
| Acceptance Repair | Opened targeted repair MR !63 |
Original frozen missions replayed at commit 20935b46 |
All 6 criteria PROVEN |
| Applicable mission runs after repair | 7 of 7 passed |
The original frozen mission bundle was replayed unchanged after the repair. The requirement stayed fixed while the implementation changed.
Explore feature MR !62 and repair MR !63 to inspect the failure, repair, and passing evidence.
Results from the demonstration
The cart-quantity example demonstrates one deliberately seeded requirement defect caught beyond the developer-written tests.
On the original feature commit, 5 of 6 issue criteria were PROVEN and AC5 was BROKEN. Developer tests passed, but the acceptance gate failed.
Acceptance Repair prepared the fix in MR !63. After human review and merge into the feature branch, ShipProof replayed the original frozen bundle unchanged: 6 of 6 criteria were PROVEN, with 7 of 7 applicable mission runs passing.
The public MR contains both verdicts, their tested commits, and downloadable evidence. These results describe this demonstration; broader effectiveness across repositories remains to be evaluated.
How the delivery loop works
1. Prove the feature before merge
GitLab CI runs the project’s developer-written tests and security checks.
Separately, the Acceptance Supervised Duo flow reads the linked issue and changed files, generates independent browser or HTTP missions, and executes them against the candidate.
Its report connects each acceptance criterion to the exact tested commit and available screenshots, video, and traces:
- PROVEN: execution supports the promise.
- BROKEN: execution demonstrates a mismatch.
- UNVERIFIED: evidence is missing, invalid, or insufficient.
The acceptance gate validates the published evidence against the current MR commit. Missing, stale, or mismatched evidence cannot satisfy it.
2. Turn failures into reviewable repairs
On an explicit request, Acceptance Repair uses the failure evidence to prepare a targeted fix on a separate branch and open a Draft MR.
The repair is checked against the frozen acceptance missions. A person reviews the proposed change and decides whether to merge it.
After the repair reaches the original feature branch, the original frozen bundle is replayed against that updated MR head. A passing repair pipeline alone does not establish that the original feature MR is ready.
3. Prove the deployed candidate
After an approved merge, the eligible protected-main pipeline runs four release steps:
- Deploy: create a tagged Cloud Run candidate revision with 0% production traffic.
- Validate: check application health and deployed commit identity.
- Replay acceptance: run the relevant merged MRs’ frozen missions against the tagged candidate.
- Promote: move production traffic to the verified revision only after the required checks pass.
A failed candidate remains unpromoted. If the public-URL check fails after promotion, the release script attempts to restore the previous revision.
4. Keep checking after release
A scheduled pipeline checks production health and runs desktop and mobile shopper journeys. The desktop journey places a synthetic no-payment order.
The monitor checks the revision actually serving production traffic. It does not rebuild or redeploy the application.
Try it in 60 seconds
- Open issue #7 and read the quantity-one acceptance criterion.
- Open MR !62 and compare the BROKEN and PROVEN acceptance reports.
- Open MR !63 to inspect the targeted repair.
- Visit Northstar Store, add an item, and decrease its quantity from 1. Confirm that it disappears and checkout becomes disabled.
- Inspect the successful scheduled monitoring pipeline for the operational follow-up.
No account, payment details, or credentials are required to try the storefront.
How we built it
GitLab Duo Agent Platform powers three flows: Acceptance Supervised, Acceptance Repair, and Incident Triage. Their definitions and instructions are versioned in the repository, with enabled flows and triggers configured in GitLab.
Node.js scripts provide mission validation, evidence publication, reporting, and GitLab integration.
Playwright executes browser missions and captures supporting evidence. Frozen mission bundles preserve the journeys across repair verification and deployed-candidate replay.
GitLab CI/CD coordinates developer tests, dependency auditing, evidence gates, packaging, deployment, and traffic promotion.
Google Cloud Run provides tagged candidate revisions and revision-level traffic control. GitLab authenticates to Google Cloud using OIDC and Workload Identity Federation, without a stored service-account key.
Agents reason about requirements and failures. Deterministic checks enforce evidence identity and release conditions.
Autonomy level: Supervised
Duo flows generate and execute missions, investigate failures, and prepare repair proposals. People review and approve merges.
After an approved merge, CI deploys and promotes the candidate automatically when its configured checks pass.
The Duo flows do not merge changes, deploy revisions, or promote traffic. Those release actions belong to the guarded CI/CD pipeline.
DevSecOps lifecycle coverage
| Stage | ShipProof capability |
|---|---|
| Plan | Linked issue criteria and repository-aware execution planning |
| Create | Draft repair MRs based on failure evidence |
| Verify | Developer tests, independent missions, and deployed-candidate replay |
| Package | Checksummed application and toolkit archives, plus Cloud Run source build |
| Secure | Dependency auditing, configured advisory review, and scope guardrails |
| Release | Candidate deployment, verified traffic promotion, and protected-tag toolkit release |
| Configure | Repository adapters, regression configuration, and keyless GCP authentication |
| Monitor | Scheduled health checks, shopper journeys, and incident issues |
| Govern | Commit-bound evidence gates, pipeline audit, and human merge review |
These are lifecycle areas reached by the combined flows and pipeline jobs, rather than nine separate agents.
Challenges we faced
Distinguishing product failures from generated-test defects
An invalid locator does not prove the application is broken. We needed explicit uncertainty, inspectable execution results, and controlled plan replacement without weakening the issue’s requirements.
Keeping evidence attached to the correct change
Original MRs, repair MRs, and deployed revisions have different identities. Safe replay requires checking the project, MR, commit, issue criteria, and mission bundle.
Connecting agent sessions with CI
Duo sessions and pipelines execute separately. We built evidence publication and validation so CI could consume verifiable results rather than rely on an agent’s success message.
Replaying proof after deployment
Local verification cannot establish that the correct application was deployed. Candidate replay checks the frozen missions against the live revision before production traffic is promoted.
Accomplishments we’re proud of
We demonstrated an inspectable path from a missed issue promise to a verified repair and guarded release.
The independent mission caught a boundary case outside the developer tests. Acceptance Repair proposed a focused change. Human review controlled the merge, and the original frozen missions passed against the updated feature commit.
Reviewers can follow the requirement, tested commit, execution result, repair, and supporting artifacts through GitLab.
What we learned
Independent testing earns trust when its conclusions are traceable and reproducible. A useful report explains what passed, what failed, and what remains unknown.
Safe automation also needs clear stopping conditions. Missing evidence stays unverified, a failed candidate stays unpromoted, and a proposed repair stays reviewable.
Useful autonomy comes from combining agent reasoning with enforceable boundaries and visible human decisions.
Current limits
The included blocking browser configuration targets Northstar’s storefront. Other repositories require a reviewed adapter, change scope, and revision-aware target. Generated missions cannot guarantee exhaustive coverage.
What’s next for ShipProof
- Validate reuse in a second repository and a different application stack.
- Measure verdict accuracy, UNVERIFIED outcomes, execution time, and cost across more changes.
- Make the existing red/green replay easier to compare directly in the evidence report.
- Turn reproduced incidents into lasting regression missions.
- Improve exploration of requirement-supported edge cases while preserving reproducible evidence.
Log in or sign up for Devpost to join the conversation.