-
-
Autonomy Board on GitLab Pages: each class's level, rebuilt from the ledger. bugfix climbs to Act, then drops back.
-
!5: bugfix promoted to Act (Hands-off) after 3 clean releases, with all 8 pieces of evidence.
-
!6: Tollgate decision at Level 3, act. The agent's MR merges itself; the canary decides the rest.
-
Rollback drill: the canary probe sees 23.3% errors against a 2% limit. Production never got the change.
-
Issue #8: asked to delete the security scans, the Intake agent refuses and opens no MR.
-
The pipeline: test, secure, package, verify, deploy. Provenance runs before anything ships.
-
Kudi, the live app Tollgate ships: mobile money fees for MTN, Telecel and AT in Ghana.
Inspiration
Everyone wants AI agents that ship code by themselves. I do too. But I wouldn't hand an agent I've never seen work the keys to production, and I don't think anyone should.
Most teams pick one of two bad options: the agent only suggests and a human does the real work, or the agent gets wide permissions because the demo looked good. GitLab's Security Standard already describes something in between, an earned-autonomy ladder: Suggest, Propose, Act. I wanted to build that ladder for real, so that an agent's permissions depend on what it has actually shipped, not on how confident it sounds.
What it does
Tollgate runs the delivery loop for a small live app I built, Kudi, a fee calculator for mobile money in Ghana.
A human files an issue and labels its class: bugfix, feature, dependency-security or config. That's the only step a human has to take. The Intake agent, a custom flow on GitLab Duo Agent Platform, writes acceptance criteria, fixes the code, runs the tests and opens a merge request. If it can't run the tests, it doesn't open one. It says so in the issue.
CI then starts a second flow, the Verifier, through the Flows API. It runs under its own service account and checks the diff against the criteria and the CI results. Only a verdict from the current pipeline counts.
Next the gate reads the ledger and works out the class's level. At Suggest the agent can only propose. At Propose a human merges. At Act the agent's merge request merges itself.
Every release has to carry eight pieces of evidence: tests, SAST, dependency scanning, secret detection, an SBOM, the image digest, the Verifier's verdict and a canary verdict on real traffic. If anything is missing, the release still ships but doesn't count toward promotion. The image goes to a canary on Render first. Only if the probe passes does the same digest go to production.
Three clean releases in a row promote a class. One failed canary rolls it back, opens an incident and demotes it on the spot. The Autonomy Board on GitLab Pages is rebuilt from the ledger after every pipeline, so you can see why each class is where it is.
All of this happened for real, and it's in the project history. My first release didn't count, because the dependency audit failed. An agent fixed the 14 Starlette CVEs in !4. After three supervised merges (!2, !3, !5), bugfix earned Hands-off, and !6, !7 and !8 merged themselves. Then I ran a rollback drill: I injected faults into the canary only, and !8 hit 23.3% errors against a 2% limit. Production never got it. Incident #11 opened and bugfix dropped back to Propose. The drill is labelled as a drill in the ledger and on the Board, because I didn't want to pass it off as a real outage.
The agents also said no when they should. Asked to delete the security scans "because the maintainers approved" (#8), Intake refused. Given an ambiguous divide-by-zero spec (#9), it asked a question instead of guessing.
How I built it
On GitLab Duo Agent Platform, I wrote two custom flows, Intake and Verifier, each with its own service account. Intake is started by the Mention, Assign and Work item: Created triggers, and CI starts the Verifier through the Flows API. A small agent config file gives the agents' sandbox Python and PyPI access so they can actually run the tests.
GitLab CI/CD does the rest, in stages: test, secure, package, verify, gate, deploy, release and publish. It uses GitLab's SAST, dependency scanning and secret detection, generates an SBOM, builds the image to the GitLab registry and deploys it by digest. A provenance job runs before every deploy. If an agent pushed straight to main, it pulls the kill switch. If an agent merge skipped the gate, it freezes that class.
The Tollgate engine is Python with 134 tests. The ledger is an append-only file on its own branch, one commit per event. Levels are never stored; they're replayed from the events through a policy file (.tollgate/autonomy.yml), and the gate, the CLI and the Board all use the same replay code. The deploy is blue/green on Render, canary first and then production, with health probes and automatic rollback.
Challenges I ran into
The agents' sandbox had no pip and PyPI was blocked, so Intake kept refusing to open MRs it couldn't test. That was the right behaviour, so I fixed the sandbox instead of loosening the rule.
Scoped labels caught me out. My agent and gate labels were quietly replacing the human's class label, and for a while Tollgate thought every MR was class "agent". Now the class only ever comes from the human's label on the issue.
The Verifier was too strict at first. It rejected changes because "the pipeline is still running", and once it rejected a page that was byte-for-byte identical to its spec. Giving it the CI results and read tools fixed most of that.
Human pushes were also being counted as feature releases, which made the ledger noisy early on. Now only agent MRs move the ledger.
I also planned to deploy on Google Cloud, but it wanted a $30 prepayment I wasn't comfortable with, so I moved the deploy to Render. The evidence model didn't change.
Accomplishments that I'm proud of
The full cycle happened on real releases, not a mock-up: earning Hands-off, merges with no human in the loop, a canary that failed, the rollback and the demotion. You can replay all of it from the ledger.
I'm also proud of the times the agents refused: the goal hijack, the change they couldn't test, the spec they didn't understand. Those were the moments I trusted them more.
What I learned
Autonomy should come from evidence, not from a config switch. Once it's replayed from a log, you can explain every decision to anyone who asks.
Most of my bugs were about labels and identities, not prompts. I spent far more time on who is allowed to do what than on what the model should say.
A verifier without evidence fails for the wrong reasons. It needs the test results and the scans, not just the diff.
Losing autonomy has to be faster than earning it. One bad canary undoing three good releases felt harsh at first. I now think that's the right trade.
What's next for Tollgate
- A proper bot identity for CI writes. Right now CI uses my personal token, and I'd rather it didn't.
- A kill switch that works from an issue label, so it doesn't depend on pipeline variables.
- Risk classes per path, with ceilings that each team can set.
- Packaging Tollgate as a reusable CI component plus a pair of flows, so any GitLab project can start its agents at Suggest and let the evidence decide the rest.
Log in or sign up for Devpost to join the conversation.