ShipCheck

Inspiration

AI made writing code cheap, but the work after it didn't shrink. On a small team, the same one or two people review every MR, chase red pipelines, check for leaked keys, remember the changelog, write release notes, deploy, and then check production. None of that is hard. It's just constant and full of interruptions, so it's a good fit for agents, as long as people keep control of what reaches production.

What it does

ShipCheck is a set of GitLab Duo Agent Platform custom flows, each started by a GitLab trigger, that cover the path from issue to production:

• Issue Triage (work item created / mention): labels type, priority and risk; finds duplicates; proposes testable acceptance criteria.
• MR Ship Review (MR created or ready / reviewer assigned): a supervisor agent delegates to code-review, security and compliance sub-agents, then posts one readiness report (verdict, stage-by-stage table, must-fix items with file:line) and labels the MR ship::ready, ship::needs-changes or ship::blocked.
• Pipeline Doctor (pipeline failed): reads the failing job logs, classifies the root cause (code bug, test bug, lint, CI config, policy gate, security finding, flaky/infra) and opens a minimal fix MR, or files an issue when a person has to decide.
• Release Scribe (mention: "prepare release 0.2.0"): collects MRs merged since the last tag, writes Keep-a-Changelog notes by label, bumps the version and opens the release MR. CI then tags, creates the GitLab Release and publishes the package.
• Deploy Watch (pipeline passed/failed): after the production deploy it reads the post-deploy smoke test, posts a deployment summary on the merged MR, and opens or closes incident issues with rollback steps.

A single policy file (.gitlab/duo/ship-policy.yml) defines what "shippable" means. The deterministic ship-gate CI job enforces it, and the agents quote that evidence instead of guessing. People approve two things only: the merge and the production deploy (a manual job). Every agent action leaves a note, label, MR or issue, so the audit trail stays in GitLab.

Lifecycle coverage: plan (triage), create (fix MRs, release MRs), verify (tests, Pipeline Doctor), package (wheel + container image + PyPI registry), secure (SAST, secret detection, security sub-agent), release (Release Scribe, tags, GitLab Releases), configure (agent-config.yml, CI lint, environments), monitor (smoke tests, Deploy Watch, incidents), govern (policy gate, CODEOWNERS, compliance sub-agent, audit trail).

How we built it

• GitLab Duo Agent Platform: 5 custom flows written in the flow registry v1 YAML schema (AgentComponents with local prompts, supervisor mode with sub-agents, least-privilege toolsets, coding_environment: none for API-only flows), triggers (MR, pipeline events, mention, work item), a custom chat agent ("Ship Captain"), AGENTS.md, project Agent Skills (skills/*/SKILL.md), agent-config.yml, and custom review instructions for the foundational Code Review flow.
• GitLab CI/CD: lint → test → secure → ship-gate → package → release → deploy → monitor, with JUnit and coverage reports, SAST and Secret Detection templates, the container registry, the PyPI package registry, the release keyword, a GitLab Pages production environment behind a manual approval, and a post-deploy smoke test.
• Sample app: ledgerlite, a small Python invoice library, CLI and stdlib HTTP API with strict money rules (Decimal, half-up cents, discount before tax), so the agents have real rules to enforce.
• Flows as code: tools/validate_flows.py checks every flow against the documented custom-flow rules (allowed fields, known tool names, router targets, reachability, sub-agent descriptions, prompt variables bound to inputs, size limits) in the lint job. 71 tests cover the app, the ship gate, the release tooling and the validator.

Challenges we ran into

• Making agents useful without making them noisy. We moved from one comment per agent to one consolidated report from a supervisor, and capped inline notes at five.
• Keeping agents honest. Instead of letting them judge "is it ready?" from scratch, a deterministic gate produces the evidence and the agents explain and act on it.
• Safe autonomy: flows get only the tools they need, never merge, never deploy, never weaken tests or add allow_failure. GitLab's rule that service-account actions don't fire triggers prevents flow loops.
• Each trigger type hands the flow a different goal (IID, mention text or webhook payload), so every prompt handles all the formats its triggers can send.

Accomplishments that we're proud of

• An end-to-end path, issue → review → auto-fix → merge → deploy → verify → release, that touches all nine DevSecOps stages.
• A planted-bug demo MR that the system catches four ways (boundary bug, float money, fake secret, missing changelog) and partly repairs on its own.
• Flow definitions are validated in CI like any other code.

What we learned

Agents work best as reviewers and fixers that stay grounded in deterministic CI evidence. Supervisor mode with focused sub-agents gives clearer reports than one large prompt. And "supervised" autonomy (approve outcomes, not steps) is a good fit for teams that want speed without giving up control of production.

What's next for ShipCheck

• Auto-remediation for dependency and SAST findings, using the vulnerability tools.
• Canary deploys, with Deploy Watch reading OpenTelemetry signals from GitLab Observability.
• Packaging the flows as public AI Catalog items so any project can enable them in a few clicks.
• A "hands-off" mode for low-risk changes (docs, chore) where a READY report auto-merges.

Built With

• docker
• gitlab
• gitlab-ci
• gitlab-container-registry
• gitlab-duo-agent-platform
• gitlab-package-registry
• gitlab-pages
• pytest
• python
• sast
• secret-detection
• yaml