SoloShip

Risk-scored merge requests with GitLab Duo Agent Platform

Comment

See it in 60 seconds. Watch the video, or open merge request 37. Nobody mentioned a bot. Opening the merge request started one custom GitLab Duo flow, and six notes appeared in order: score, review, config check, security triage, release decision, audit record. The score, 2.38, is the same number the engine prints on a laptop. GitLab Duo agent session pages may ask for a GitLab sign-in; the repository and the merge requests open without one.

Inspiration

When a team has a second reviewer, a typo fix, an auth change, and a leaked secret get different attention. A solo developer or a two-person team does not have that reviewer, so all three take the same road: merge. When code gets cheaper to produce, deciding what is safe to ship becomes the scarce skill. We wanted the missing reviewer. Not a chatbot's opinion, but a decision you can reproduce, explain, and audit.

What it does

SoloShip scores every merge request from 0 to 100 and picks one of four routes.

Band Score Route What happens
LOW 0 to 8 Hands-off Can pass every gate
MEDIUM 9 to 59 Supervised Automatic until production, then one person approves
HIGH 60 to 100, or raised by a rule Assisted A person at every gate
CRITICAL confirmed secret, score 100 Blocked The release is blocked

Security scans stay on for every band.

The score comes from a deterministic Python engine, not from a model's opinion. GitLab Duo Agent Platform flows run the engine on the real diff, read the result, and post it on the merge request.

One trigger, six steps. The custom flow soloship-pipeline starts when a merge request is created. Six components run in order: risk scorer, reviewer, config guard, security triage, release summary, and auditor. Each one reads the previous note and quotes it. On merge request 37 the flow finished with score 2.38, LOW, Hands-off. It ran again on merge request 38, once when the draft was created and once when it was marked ready.

Built on GitLab Duo Agent Platform

Duo is how SoloShip works, not a label on it.

  • Ten custom flows in the AI Catalog, each with its own service account: the soloship-pipeline orchestrator and nine single-purpose flows (risk scorer, reviewer, config guard, security triage, release summary, auditor, planner, creator, monitor).
  • Real tools, real diffs. The flows call get_merge_request, list_merge_request_diffs, and create_merge_request_note. They run in a python:3.12-slim environment with a setup script, so the engine executes inside the Duo runner.
  • Flows that do not guess. The reviewer must quote a changed line. Security triage reports a finding only when a scanner artifact contains it. When the diff cannot be read, the scorer says so and posts no score.
  • Every decision is recorded. The auditor appends one line to a hash-chained JSONL log on the audit-log branch. soloship audit verify accepted 8 records.
  • CI around it. The pipeline runs tests, pip-audit, secret detection, Semgrep SAST, a container image build, and a staging health check. A deploy_production job exists as a manual approval gate for the Supervised route. It is a gate, not a real deploy.
Flow Trigger What it does Session
soloship-pipeline Merge request created Runs all six steps in one session 9297544
risk-scorer Mention Runs the engine on the diff, posts score, band, route 9296937
reviewer Mention Quotes a changed file and a changed line 9296955
config-guard Mention Names changed paths, flags policy and sensitive ones 9296964
security-triage Mention Reports only what a scanner artifact contains 9296973
release-summary Mention Quotes the score and says whether production waits 9296984
auditor Mention Appends one record to the audit chain 9296998
planner Mention Posts the task split as an issue note 9296328
creator Mention Opens a draft merge request with a test of an existing helper 9296600
monitor Mention Posts a finding present in a successful SAST artifact 9296365

The first six mention flows ran as a chain on merge request 36. Planner, creator, and monitor each ran on their own mention.

How the score works

The engine reads soloship/risk/weights.yaml, sums ten signals, and applies five hard rules that can change the published band.

Hard rule Effect
Confirmed secret Score 100, band CRITICAL
Critical new SAST finding or critical dependency vulnerability Band at least HIGH, score unchanged
Policy file (.gitlab-ci.yml or the weights file) Band at least HIGH, score unchanged
Sensitive path (auth/, security/, migrations/, Dockerfile, .gitlab-ci.yml) LOW is raised to MEDIUM
Failed test gate Blocks the release

That is why the published band and the number can differ, and why we publish both side by side:

Merge request Score Band Why
!3 docs change 2.56 LOW Wording only
!36 auth helper 23.23 MEDIUM Score is in 9 to 59, and the auth path is on the sensitive-path floor
!23 critical SAST fixture 21.15 HIGH The security rule raises the band, the score stays 21.15
!27 confirmed secret fixture 100 CRITICAL The secret rule sets both

Challenges

  • The numeric band and the published band are different facts, and a single cutoff table hides the hard rules. We fixed this by publishing score and band together.
  • Early flows could not read merge request diffs. We added the diff tool and made the empty-diff case post "no score produced" instead of crashing.
  • Duo's default environment skips setup. We moved to a full environment with an image and a setup script so the engine runs inside the runner.
  • A config guard prompt missed waves/w17/soloship/risk/weights.yaml because it matched an exact path. We changed the prompt to match path endings and reran it.
  • The group MCP setting could not be enabled with our role.

Accomplishments

Measured results, not a user count or a time-saved claim.

Measurement Result
Frozen label sets, exact band 19/30, 11/15, 13/20
Within one band 30/30, 15/15, 20/20
High-risk merge requests labeled LOW 0 on all three sets
GitLab risk-scorer notes matching the engine 30 of 30 merge requests
Tests 101 passed locally
Install from a warm wheel cache 6.1885 seconds
One soloship score 0.1382 seconds
Duo flows with a finished session 10
Audit chain 8 records verified

The same assistant wrote all three label sets, so treat the accuracy numbers as a consistency check, not independent validation. Details are in docs/EVIDENCE.md.

Adopt it in three steps

  1. python3 -m venv .venv
  2. Install the repository with .venv/bin/pip install
  3. .venv/bin/soloship score

A second folder on the same machine did this in 6.19 seconds from a warm wheel cache.

What we learned

A flow is only useful if it reads the diff and quotes it. A deterministic core makes a language-model flow trustworthy, because the model reports and routes while the number comes from code. And an audit trail changes how a decision feels: you can verify it later.

What's next

  • Run the route's parent pipeline example on GitLab.
  • Give Package its own Duo flow. Today it is a CI image job.
  • Let Duo call the engine as an MCP tool (score_change). The server is defined, but the group flag is off, and the AI Catalog path needs an approved HTTP endpoint.
  • Start the existing nine flows from triggers other than mention.

Honest limits

  • Held-out exact band match is 11/15, short of our locked 12/15 target. The target was not changed.
  • The nine single-purpose flows ran only when a person mentioned them. The orchestrator ran from the Merge request created and marked ready triggers.
  • MCP was not enabled for this group, and no session called it.
  • deploy_production is a manual approval gate, not a real deploy. The LOW-band "automatic" path is policy, not wired into CI.
  • Package has no Duo flow. Create adds a test and does not implement the issue.
  • Software carbon intensity was not measured.

Everything above is reproducible from the public repository. Start with the video, then merge request 37.

Built With

  • fastapi
  • gitlab-ai-catalog
  • gitlab-ci/cd
  • gitlab-duo-agent-platform
  • pytest
  • python
  • yaml
Share this project:

Updates

Submission history