Ratchet

A ratchet turns one way: AI agents can raise a change's risk, never lower it. Risk sets autonomy: low ships hands-off, medium waits for one approval, high for a human review.

Comment

Inspiration

Code generation is solved. Trust is not.

Every delivery pipeline I have worked with makes one choice for every change: either a human approves each step, or nobody does. A typo fix and a change to authentication wait in the same queue, behind the same approvals. Teams that want to move fast end up switching guards off; teams that want to stay safe end up rubber-stamping.

The hackathon asks how much control you hand over. My answer: it depends on the change. Autonomy should not be a team setting. It should be decided per merge request, by looking at what that merge request actually does.

What it does

Ratchet gives every merge request a risk level, and the risk level decides how much the pipeline may do on its own:

Risk Autonomy What happens
risk::low Hands-off Merges and ships to production with no human
risk::medium Supervised Merges and ships to staging; one human approval for production
risk::high Assisted A human reviews the merge and approves production

Around that decision runs the whole post-code lifecycle. You open an issue, and agents plan it, write it, test and scan it, judge its risk, merge it, deploy it to staging, roll it out to production behind a canary, publish a release, and report back on the issue. When a release breaks production, the loop runs backwards: rollback, incident, root-cause analysis, a planned fix, and a new release that closes the incident.

In the run filmed for the demo, issue #8 ("Link to the source repository from the dashboard footer") went from a human typing it to "live in production as v0.4.1" in 15 minutes: Triage planned it after 1½ minutes, the Developer flow opened the merge request after 3½, the Governor agreed on low after 5¾, and the canary promoted it nine minutes later. The issue closed itself with the note "No human was involved: the change shipped hands-off."

Why "Ratchet"

A ratchet turns one way. So does risk here. Two independent assessments look at every commit:

  1. A deterministic check applies a policy file to the diff and the scanner reports: sensitive paths, dependency changes, diff size, untested code, security findings in the changed files.
  2. The Governor, an agent built on the GitLab Duo Agent Platform, reads the merge request and writes its own decision with a rationale.

The final level is simply

$$\text{risk}{\text{final}} = \max\left(\text{risk}{\text{rules}},\ \text{risk}_{\text{agent}}\right)$$

The agent can raise risk, never lower it. That rule lives in ordinary Python, not in a prompt, so a prompt injection hidden in a diff cannot talk the system out of a guard. Only notes from trusted accounts count, each decision is tied to one commit, a missing answer means at least medium, and any change to the Governor itself, its policy or the pipeline is always high.

The Governor earned its place on day one. On the very first live run it rated a README change high, because the security scan had found a real SSRF risk in Ratchet's own GitLab client. The rules had scored it low. I fixed the finding instead of silencing it. Later it raised an agent-written feature to medium because the developer agent admitted it could not run the tests locally, so production waited for a human.

When production breaks

The most useful test was a change that looked harmless: a badge showing "Staging" or "Production" in the dashboard header. The rules scored it low. The Governor read the diff and raised it to medium: the label came from a fragile hostname lookup with no test for production hosts. GitLab Duo Code Review asked for changes for the same reason. For the demo I overrode both on purpose and approved production.

The canary sent 80 requests to the new revision, saw 500 errors on /, and rolled back by itself. Only the canary's 10% of traffic ever saw the bad revision, for under two minutes. CI then opened GitLab incident #6 and summoned the Incident flow, which read the verification log and the diff and wrote an accurate root-cause analysis, including why the tests passed: the test client always used the staging hostname, the one shape that worked. It filed a planned fix issue, and the relay handed it to the Developer flow.

Something I did not plan happened next: GitLab's Fix CI/CD Pipeline flow had also seen the failed canary pipeline and wrote its own fix. Two agents raced for one bug. Its fix landed first; the Governor kept it at medium, one human pressed one button, and v0.3.1 shipped. Since then, any release that contains the failed commit and passes its canary closes the incident and the losing duplicate fix by itself, whichever agent wins.

How I built it

GitLab Duo Agent Platform

  • Triage (custom flow, trigger: work item created) writes a plan, acceptance criteria, an out-of-scope list and a risk estimate.
  • Developer (foundational flow) implements the plan and opens a merge request.
  • Governor (custom flow, trigger: mention posted by CI) reviews each commit and posts a machine-readable decision.
  • Incident (custom flow, trigger: mention posted by CI when a canary rolls back) writes the root-cause analysis and files a planned fix issue.
  • Pipeline Doctor: the foundational Fix CI/CD Pipeline flow, triggered when a pipeline fails, including a failed canary.
  • Code Review: automatic GitLab Duo review on every merge request.

GitLab CI/CD runs tests, builds the image, and runs SAST, secret detection and dependency scanning. Then risk-score posts the deterministic note and summons the Governor, and governor-gate waits for its decision, combines the two, labels the merge request and enables auto-merge unless the change is high risk. On main, a release-plan job reads the merged risk level and generates the production child pipeline: for low risk the canary starts on its own, for medium and high it waits for a human to press "Run job".

Google Cloud: images go to Artifact Registry, staging and production run on Cloud Run, and production ships behind a canary: 10% of traffic, 80 verification requests, then promote to 100% or roll back. There are no service account keys anywhere: CI exchanges its GitLab OIDC token for a 30-minute Google token through Workload Identity Federation, and Google only accepts tokens from protected branches. After every successful rollout, Ratchet publishes a semantic-versioned GitLab release (high risk bumps the minor version) with the risk decision and canary result in the notes.

The Ratchet dashboard on Cloud Run shows every decision: what the rules said, what the agent said, what was enforced, and why.

Challenges I ran into

  • Agents cannot wake up agents. GitLab does not start a flow from a mention, or an assignment, written by another flow. That is a sensible loop guard, but it broke the Triage-to-Developer hand-off. I built a small relay: Triage writes a hand-off marker, a project webhook sends comments to the dashboard, and the dashboard checks the webhook secret, the author and an allow list of targets before posting the mention once, as a human. The secrets live in Secret Manager; CI copies its token in, and nothing ever prints it.
  • Two agents, one bug. The Incident flow and the Fix CI/CD Pipeline flow both reacted to the same failed canary. Rather than forbid one of them, I made the resolution independent of who fixes it: the release step checks whether the failed commit is part of the release, and if so closes the incident and supersedes the other fix.
  • Prompts may not contain HTML comments, so the Governor's decision marker became a visible inline-code line, which turned out to be more transparent anyway.
  • Agents open draft merge requests, which never auto-merge. The gate now marks low and medium drafts ready, because the Governor's review is the review.
  • Small platform surprises: Cloud Run reserves /healthz, merge request notes are not readable without a token, the flow sandbox had no pip and no network until I configured it, and hackathon participants cannot create project access tokens. Each one is documented in the README.

What I learned

Guardrails belong in code, and judgment belongs in agents. The agent is good at noticing what a rule would miss: an unrun test, a suspicious finding, a sentence in a description that tries to steer it. The code is good at making sure that judgment can only ever make the system more careful.

What's next

  • Decide the production gate from every unreleased change, not only the last merged one. Today a hands-off change merged right after a high-risk direct push would carry it to production; I deploy high-risk changes on their own until this is fixed.
  • Let the Incident flow and Pipeline Doctor coordinate instead of racing, so only one fix is written.
  • Learn the risk policy from history: which paths and authors actually caused rollbacks.

Built With

Share this project:

Updates

Submission history