Shipyard

A delivery pipeline where public steers. Anyone can asks for a feature and GitLab Duo agents ship it with no human. Don't be afraid of breaking, it heals itself

Comment

Inspiration

On a small team, one person gets the call when production breaks. AI made writing code fast, but everything after the code still lands on that person: checking a change is safe, shipping it, watching it, finding and fixing what broke at night. We wanted to see how much of that life after code GitLab Duo agents could take on, safely enough that strangers could drive it.

What it does

Shipyard is a delivery pipeline the public steers. Nobody on the team starts anything: anyone on the internet can ask its site for a feature, and five GitLab Duo agent flows take it from request to production on Google Cloud, across all nine DevSecOps stages:

  1. Plan. A guard refuses prompt injection, links and secrets. The triage agent writes acceptance criteria, or declines and offers the nearest version it can build, which the visitor takes with one click.
  2. Create. The build agent writes the code and its tests; the fix agent repairs red pipelines and review feedback.
  3. Verify. Lint with Code Quality, tests with coverage, fuzz tests, and page-weight and prompt-token budgets. The review agent checks every acceptance criterion.
  4. Package. Images are signed keylessly with Sigstore, with a signed SBOM attestation, both verified before every deploy.
  5. Secure. SAST, IaC scanning, secret detection, dependency and image scans and a security gate on every change, plus nightly DAST and HTTP fuzzing of staging.
  6. Release. Staging, a canary, a 60-second watch, then production, one deploy at a time, and a GitLab Release that records who built it.
  7. Configure. All of Google Cloud in Terraform with GitLab-managed state; traffic is moved by code during incidents; a nightly check catches configuration drift.
  8. Monitor. A synthetic probe opens an incident within about 30 seconds and rolls traffic back. A production bisect names the exact commit, then the SRE agent chooses a revert or a small fix and files a regression test. Every resolved incident becomes a lesson in the instructions of every agent that writes, reviews or repairs code.
  9. Govern. Only the review agent's own account can approve (a forged approval is removed in seconds), agents are fenced off the controls, least-privilege cloud permissions are checked by a test, and licences and agent budgets are enforced.

After launch, the person who asked says whether the feature is right. "Not quite" goes straight back to the agents.

The record, not the best runs: 32 requests from 16 to 25 October, every one counted. 21 went live, with a median of 15 minutes from request to live (13.8 minutes since the system was complete). A person stepped in three times, each a tidy-up after a drill or a false alarm.

The final drill was pre-registered: we committed a hash of the plan, then planted a bug that passed all 364 tests. Production failed, and the incident opened 31 seconds after the fault. The bisect named the commit, and the agent's revert was live 10 minutes after the outage. No person acted.

How we built it

  • Five custom Duo flows (triage, build, review, fix, SRE), public in the AI Catalog and versioned in the repository. A custom Duo Chat agent, the Shipyard Concierge, also files requests, and the ops service is an MCP server for Duo Chat and the Duo CLI.
  • A dispatcher with no AI in it. A small program on Cloud Run watches GitLab webhooks and Cloud Monitoring alerts, decides in plain code what happens next, and starts the next flow through the Duo Flows API. It merges only at the commit the review agent approved, and posts a release decision record.
  • AI makes the judgement calls; code makes the safety calls. Agents may change the product, never the controls. This is enforced three times: a CI scope policy, the dispatcher re-checking every merge from main, and a protected test of how public input is fenced.
  • GitLab CI/CD runs the tests, fuzzing, SAST, secret detection, dependency and container scans, a security gate, Code Quality and licences, plus nightly DAST, an HTTP fuzzer and a configuration drift check against the running system.
  • Google Cloud: Cloud Run, Workload Identity Federation (no keys), Secret Manager and Cloud Monitoring, all in Terraform with GitLab-managed state.
  • GitLab is the only database: requests are issues, and each request's stage is a label.

Challenges we ran into

This was my first time doing DevSecOps. Before I could build anything, I had to learn how it all fits together: pipelines, security scanning, signing, deploys, monitoring and incident response. Much of the first stretch was reading and experimenting, not building.

Then running the system for real found 18 defects that tests never would. They are all in docs/BUGS.md, with what changed:

  • GitLab's agent service dropped a session with no output; the dispatcher now retries a lost session as soon as it ends.
  • Three requests at once exposed a deploy race; each environment now deploys one change at a time, newest first.
  • A deploy GitLab refused as outdated was misread as an outage, which opened a false incident.
  • A merge waited for a webhook that never came; it now follows the approval in the same step.

Each one is fixed and tested, and the three-at-once run passed cleanly on its second try.

Accomplishments that we're proud of

  • Finishing a system that covers all nine DevSecOps stages, even though this was my first time doing DevSecOps.
  • A stranger's request reaching production in about 15 minutes, with nobody in between.
  • A real outage healed in 10 minutes, with the cause proven by an experiment before any AI touched it.
  • Triage that negotiates instead of just saying no.
  • A carbon receipt on every feature (13.1 g CO2e on average, by the SCI method).
  • Every claim in the README links to the run that proves it, with a 10-minute route for judges to check them.

What we learned

Trust comes from where you put the line, not from the model. Agents are good at judgement: what to build, how to fix it, whether it meets the criteria. Code is better at authority: who may approve, what may merge, when to roll back. And running a system for a week teaches more than any amount of testing.

What's next for Shipyard

  • Bring a reverted feature back automatically, with its fix.
  • Run a merge train, so changes that pass alone but fail together are caught before they merge.
  • Let teams set their own rules for what triage accepts. docs/ADOPT.md already takes the automation to any project in three levels.
  • Package the pipeline as a GitLab CI/CD Component, so any project can include Shipyard's checks and agent chain with one line.

Built With

  • fastapi
  • gitlab
  • google-cloud-run
  • python
  • terraform
Share this project:

Updates

Submission history