Ratchet

Inspiration

During an outage, nobody opens a merge request. Engineers fix production directly: kubectl edit a memory limit, kubectl scale a deployment. Then one of two things happens. GitOps quietly reverts the fix and the outage comes back, or the fix never reaches Git, the cluster and Git disagree, and nobody asks why the incident was possible in the first place. Worse, not every change made in a hurry is a fix. Some just open a security hole.

The tools we have today can see the drift, and Flux can undo it, but neither one learns anything. I wanted infrastructure that gets stronger after every incident, so the same incident can't happen twice. That's where the name comes from: a ratchet only turns one way.

What it does

Ratchet is self-hardening infrastructure built on the GitLab Duo Agent Platform. Every change made outside Git becomes one of two things:

• An emergency fix gets kept. The agent commits exactly the running value to Git and adds a Prometheus alert that would have caught the problem earlier.
• An unauthorized change gets reverted. The agent writes a Kyverno policy that blocks the same change from ever being applied again.

The core principle: the LLM proposes, deterministic code decides. No agent output reaches the cluster without passing non-AI CI gates, and every decision leaves a record on a GitLab incident and a merge request.

In the live run shown in the video, the only human action was a single kubectl patch that made a container privileged, with no incident behind it. From there, with nobody clicking anything:

1. drift-sentry detected the change, paused Flux for that one object, and filed GitLab incident #13 with the evidence.
2. It started the Ratchet flow through the Flows API. The triage agent ruled the change unauthorized, and the contain agent wrote a Kyverno policy and opened MR !6.
3. 86 seconds after the change, drift-sentry restored the Git values.
4. The CI gates proved the policy: guard (this MR may only add a policy), proof (the Kyverno CLI replays the original change: it must be rejected, each of its changes on its own too, while every manifest in Git is still admitted), and regress (every past guardrail still catches its incident).
5. GitLab auto-merged the MR and Flux synced the policy to the cluster.

16 minutes from the change to an enforced guardrail, with zero human actions after the change. Running the same command again gets denied at admission.

The emergency path ran hands-off the same way: when chama-ledger was OOMKilled and an engineer raised the memory limit, Ratchet kept the fix in Git and added a memory alert that promtool proved fires 34 seconds before the crash.

See it all on one page: the guardrail ledger is a GitLab Pages site that every main pipeline rebuilds. It shows each guardrail with its incident, its merge request and a fresh run of its proof.

The gates, measured: a gate scoreboard runs 19 seeded guardrails through the real gates on every pipeline. 15/15 bad guardrails are rejected (CI edits, extra replicas, swapped images, late or noisy alerts, too broad, too narrow and audit-only policies) and 0 of 4 good ones are blocked.

The ratchet, shown: draft MRs !7, !8 and !9 each weaken a past guardrail: a less sensitive alert, a narrower policy, a policy switched to Audit. regress fails all three.

How I built it

• drift-sentry is a Go controller (controller-runtime) that runs next to Flux. Kubernetes managedFields show which manager changed which field, and the Git revision Flux applied decides whether it's really drift. It pauses Flux per object with the kustomize.toolkit.fluxcd.io/reconcile: disabled annotation, files the incident, starts the flow, arms auto-merge, and keeps its state in annotations, so it survives restarts.
• The Ratchet flow (.gitlab/duo/flows/ratchet.yaml) on the GitLab Duo Agent Platform has five components: triage, codify, guardrail, contain, and escalate (a fallback that leaves Flux paused and hands the decision to a human; triage prefers "unauthorized" when unsure, so it hasn't run live yet).
• The ratchet CLI provides the deterministic gates guard, proof, and regress. It uses promtool to replay incidents against alert rules and the Kyverno CLI to replay changes against policies. ratchet scoreboard runs seeded guardrails through those gates, and ratchet ledger builds the GitLab Pages site.
• A pipeline execution policy in a separate project injects the gates into every ratchet/incident-N merge request and builds the checker from main, never from the branch under review. MR !3 shows a branch deleting the gates from .gitlab-ci.yml: the policy job still ran and blocked it.
• The demo service, chama-ledger, is a Go + PostgreSQL app for a chama (a Kenyan savings group), deployed on a kind cluster with Flux, Kyverno, and Prometheus.

Challenges I ran into

• Telling drift from deployment. managedFields say who touched a field, not whether that's wrong. The breakthrough was treating the Flux source artifact as the source of truth and using managedFields only to nominate candidate fields.
• Stopping the agent from grading its own homework. A merge request pipeline runs the branch's own .gitlab-ci.yml, so an agent could edit the gates away or rewrite the checker. Moving the gates into a pipeline execution policy that builds the checker from main closed that hole.
• Pausing just one object. Suspending a whole Flux Kustomization would freeze unrelated deploys mid-incident. Per-object reconcile annotations keep everything else syncing.
• Unattended means nobody to press retry. A dropped download from the Go module proxy once stalled an unattended merge. I added a proxy fallback and retries limited to infrastructure failures. A failing gate is deterministic and is never retried.
• Proofs that are actually proofs. An alert that "looks right" isn't enough. Each guardrail has to replay the original incident and catch it, and keep catching it on every future pipeline.
• Testing the ratchet found a hole in it. When I tried to weaken incident 13's policy on purpose, regress caught a less sensitive alert but let two policy weakenings through. A policy that blocked only one of the two fields still rejected the combined change, and the Kyverno CLI reports violations even for audit-only rules. Now each change is also replayed on its own, and every rule must enforce. MRs !8 and !9 show both weakenings failing.

Accomplishments that I'm proud of

• A fully hands-off loop, from a kubectl change to a merged, enforced, proven guardrail, recorded live with no edits to the run itself.
• Guardrails that only get stricter: regress re-proves every past incident on every pipeline, so the system can't quietly forget a lesson.
• An AI agent that can't switch off its own checks.
• Deterministic gates I can put a number on: 15/15 bad guardrails rejected and 0 of 4 good ones blocked, re-checked on every pipeline and published on the guardrail ledger.

What I learned

Agentic automation is most trustworthy when the agent's job is narrow and its output is checked by boring, deterministic code. Letting the model propose and making CI decide gave me the speed of an agent without trusting it blindly. I also learned how much the GitLab platform already provides for this: incidents, the Flows API, merge request auto-merge, pipeline execution policies and GitLab Pages fit together into a complete post-code lifecycle.

What's next for Ratchet

• More guardrail types: resource quotas, NetworkPolicies, and SLO-based alerts.
• Demonstrating the escalate path end to end, with a human approval step in the incident.
• Packaging drift-sentry as a Helm chart and the policy project as a reusable template, so any GitLab + Flux team can adopt Ratchet in an afternoon.

Built With

• claude
• docker
• gitlab
• golang
• yaml