-
-
Failed HIGH-risk CI pipeline
-
Human approval required
-
Approval completed, pipeline passed
-
Repaired pipeline — 157 passed
-
LOW-risk pipeline — 30 tests
-
Checkout bug — 5 failed tests
Inspiration
Modern software teams face a difficult challenge: how can they release changes quickly without sacrificing safety?
A small documentation update and a critical checkout calculation change should not require the same level of testing. Yet traditional CI/CD pipelines often treat every change similarly, wasting time on low-risk updates while potentially allowing dangerous defects to reach production.
We wanted to build an AI-powered release safety engineer that understands the risk behind code changes, chooses appropriate verification steps, identifies defects and keeps humans in control of important decisions.
That idea became ProofShip 2.0 — AI-Powered Risk-Adaptive Release Safety Engineer.
What it does
ProofShip 2.0 combines GitLab Duo AI with a deterministic safety policy and GitLab CI/CD to make software release verification smarter and safer.
When a developer opens a merge request:
- AI Risk Assessment: A custom GitLab Duo flow examines the code changes, identifies affected components and classifies risk as LOW, MEDIUM, HIGH or CRITICAL.
- Risk-Adaptive Testing: A Python policy controller validates the AI recommendation against the actual commit and selects an approved test profile.
- Automated Security: Bandit, Semgrep SAST and GitLab Secret Detection provide mandatory security checks.
- AI-Assisted Repair: GitLab Duo identifies potential defects and suggests fixes for human review.
- Fail-Safe Enforcement: Missing, outdated or unverifiable AI evidence triggers a conservative CRITICAL-risk profile.
- Human Approval: A blocking manual approval gate prevents the simulated release stage from continuing without authorization.
ProofShip demonstrated two real scenarios:
LOW-risk documentation change: GitLab Duo classified a documentation-only merge request as LOW risk, and ProofShip successfully executed 30 smoke tests.
HIGH-risk checkout defect: We deliberately introduced a pricing bug into MiniShop, causing five automated tests to fail. GitLab Duo identified the incorrect calculation and proposed restoring multiplication. After a developer applied the suggested fix, all 157 tests passed.
How we built it
We built ProofShip 2.0 using Python, FastAPI, GitLab Duo Agent Platform and GitLab CI/CD.
MiniShop API: We developed a lightweight FastAPI application containing health, product and checkout endpoints, supported by automated unit, boundary and regression tests.
GitLab Duo integration: We configured a custom ProofShip Risk Review flow to inspect merge requests and post AI-generated assessments with the corresponding commit SHA, risk reasons and recommended testing profile.
Deterministic safety controller: We implemented Python scripts to validate trusted AI results, enforce minimum risk levels and prevent AI recommendations from bypassing mandatory security checks.
Adaptive CI/CD: GitLab pipelines automatically execute the selected pytest profile alongside Bandit, Semgrep and secret detection.
AI-assisted debugging: We demonstrated a real failed-to-passed workflow using a controlled checkout pricing defect and a GitLab Duo-proposed repair.
Supervised approval: We implemented and successfully demonstrated a blocking manual approval gate with a machine-readable approval record.
We also added Docker packaging files for future deployment workflows.
Challenges we ran into
Integrating AI decisions with CI/CD: GitLab Duo reviews initially existed as merge request comments, while the CI pipeline needed trustworthy structured risk information. We implemented a validation process that checks the review source and exact commit SHA.
Missing merge request context: Our initial branch pipeline could not determine the merge request ID, causing ProofShip to correctly fall back to CRITICAL risk. We configured merge request pipelines to resolve this.
Security vulnerabilities: Our security checks identified XML parsing and potential server-side request forgery risks in supporting scripts. We corrected these issues and verified the fixes with automated tests.
Validating AI-assisted repairs: After correcting the checkout defect, the final aggregate diff no longer showed the original code change. GitLab Duo conservatively escalated the assessment to CRITICAL because it could not independently verify the repair from that diff alone.
These challenges helped us build a more reliable system in which AI recommendations remain subject to deterministic verification.
Accomplishments that we're proud of
- Built and executed a real custom GitLab Duo risk-review flow.
- Demonstrated LOW-risk adaptive testing with 30 passing smoke tests.
- Detected a real checkout pricing regression with five failing tests.
- Used GitLab Duo to identify the defect and propose the correct repair.
- Achieved 157 passing automated tests after applying the repair.
- Integrated Bandit, Semgrep SAST and secret detection.
- Implemented fail-closed behavior for missing or unverifiable AI evidence.
- Demonstrated a blocking manual approval gate with an auditable approval artifact.
- Published a documented, MIT-licensed GitLab repository with reproducible pipeline evidence.
What we learned
We learned how to connect AI-generated code reviews with real CI/CD execution, design risk-adaptive testing policies, validate AI outputs, implement security scanning and coordinate software development through GitLab merge requests.
Our biggest lesson was that AI should improve engineering decisions without being allowed to bypass essential safeguards.
We also learned the importance of using real test results, commit-specific evidence and human oversight rather than treating an AI recommendation as unquestionable approval.
What's next for ProofShip 2.0
Our next goal is to extend ProofShip into a complete deployment and recovery platform.
Planned improvements include Google Cloud Run staging and production deployments, secure cloud authentication, automated health monitoring, traffic rollback to previously healthy revisions and comprehensive cloud release reports.
These cloud capabilities were outside the scope of our submitted demonstration because they required additional billing setup. The current implementation focuses on the working GitLab Duo, risk-adaptive CI/CD, security verification, AI-assisted repair and supervised approval workflow.
Our long-term vision is to make intelligent release safety accessible to development teams of all sizes.
Log in or sign up for Devpost to join the conversation.