ShipLoop

AI writes the change. ShipLoop proves whether it can ship.

Comment

Inspiration

AI can write code faster than small teams can safely review, release, and operate it. The difficult part now begins after the code is generated: proving that the change satisfies its requirements, deciding whether it is safe to merge, observing what happens in production, and recovering when reality disagrees.

ShipLoop gives small teams an evidence-driven reviewer, release engineer and production responder built on the GitLab Duo Agent Platform.

Its rule is simple: no evidence, no autonomy.

What it does

ShipLoop is a hands-off GitLab Duo flow for reviewing, repairing, shipping, observing, and recovering AI-written changes.

A reviewer assignment, merge request pipeline, broken main pipeline, or production incident can start the flow. Six specialist roles then collaborate:

  • The incident commander selects the correct response.
  • Triage connects production telemetry to the affected code.
  • The reproducer creates a test that fails on the old code.
  • The fixer makes the smallest root-cause change.
  • The gatekeeper evaluates production evidence, proof tests, scope, and pipeline security.
  • The narrator records the full timeline and closes an incident only after production verification.

A change is merged hands-off only when its evidence ladder is complete. A green pipeline alone is never treated as proof.

Every review starts with a reading list. The evidence gate runs the change's tests under coverage and records which proof test ran each changed line. Proven lines are listed as proven; only the rest goes to the human reviewer, linked line by line and marked inline in the merge request diff as a GitLab Code Quality report. In MR !35 the reviewer had to read 5 of 17 changed lines. Across this repository's agent merges, 68 of 83 changed lines were proven.

How we built it

ShipLoop uses a custom GitLab Duo supervisor flow with pipeline and reviewer-assignment triggers.

GitLab CI/CD runs:

  1. Unit and regression tests
  2. A deterministic evidence gate (red on the old code, green on the change, no regressions, no skipped tests, blast radius, reading list)
  3. GitLab SAST
  4. Secret detection
  5. Dependency scanning
  6. CycloneDX SBOM packaging
  7. AWS Lambda staging deployment
  8. A 20% production canary
  9. Production verification and GitLab Release creation

GitLab Observability provides production traces and error fingerprints. Scheduled production-watch pipelines decide whether to open, resume, reopen, or verify an incident. A bad deployment can be tested against the previous Lambda release and rolled back when that version answers the failing request.

Issues and labels hold incident state. Merge requests contain the proof and autonomy decision. GitLab Pages publishes a public Incident Ledger linking every incident to evidence across the nine DevSecOps lifecycle stages.

Evidence from real runs

This is a running system rather than a proposed architecture.

In one AI-style merge request (!35), the existing test suite was green even though two of five acceptance criteria were broken and two branches were untested. ShipLoop found the gaps, added the missing proof, repaired the change, merged it, passed a production canary, and shipped it in 11 minutes without another human action.

Production has also reopened an incident (#6) when the same error returned with a second root cause. In another run (#3), ShipLoop replayed the failing request against the previous release and rolled production back.

Fifteen merge requests have been merged by the flow's service account.

ShipLoop also demonstrated when it must stop. It found and tested a valid preventive sibling fix (!36) with 115 passing tests and a green security pipeline. Because production had never observed that defect, the gatekeeper refused to merge it and marked it for human review.

Testing beyond the demo

We evaluated the same gate on 173 real AI-written Django patches from SWE-bench (SCALE.md).

  • Green CI's selections were correct 50% of the time.
  • ShipLoop's red-on-old, green-on-change evidence gate reached 74%.
  • 12 of the 65 patches the gate accepted carried files no test runs (one was a 1,318-line backup copy of a Django module). Green CI passed all of them; the reading list puts them first.
  • A preregistered benchmark produced 12/12 correct autonomy decisions, including four unsafe changes with fully green pipelines.
  • Production schema fuzzing found six previously unknown crashes; the loop fixed five hands-off and held one when its fix conflicted with main.

74% is not enough to merge on tests alone. That is why ShipLoop combines test evidence with scope analysis, security scanning, canary deployment, production telemetry, verification, and rollback.

Challenges

The hardest problem was defining when an AI agent had enough evidence to act autonomously.

We learned that successful tests are only one part of the decision. Tests can miss acceptance criteria, fail to execute changed files, or prove behavior unrelated to the production failure. We therefore built an explicit evidence ladder and made every autonomy decision inspectable.

Another challenge was preserving attribution between human and agent actions. Every human action is documented, while agent commits, merge requests, comments, labels, pipelines, deployments, and releases remain visible through the GitLab service account and public ledger.

What we learned

Reliable agentic DevOps needs deterministic gates around probabilistic agents. Agents are useful for investigation, repair, and explanation, while GitLab pipelines and production telemetry should determine whether the evidence is complete.

We also learned that the best autonomous system is one that can explain why it acted and confidently stop when a required proof is missing.

What's next

Next, we would adapt the evidence model to larger repositories, add service-specific SLO policies, and measure how much reading the reading list saves on real teams' merge requests.

For this submission, the complete source, live service, production incidents, merge requests, pipelines, releases, benchmarks, and human actions are publicly inspectable. Judges can start with JUDGING.md: eight claims, one link each, about 10 minutes.

Built With

  • amazon-web-services
  • aws-lambda
  • coverage-py
  • cyclonedx
  • fastapi
  • gemnasium
  • gitlab
  • gitlab-ci
  • gitlab-dependency-scanning
  • gitlab-duo-agent-platform
  • gitlab-observability
  • gitlab-package-registry
  • gitlab-pages
  • gitlab-releases
  • gitlab-sast
  • gitlab-secret-detection
  • httpx
  • hypothesis
  • markdown
  • opentelemetry
  • pytest
  • python
  • schemathesis
  • semgrep
  • uvicorn
Share this project:

Updates

Submission history