Earned Autonomy

Every merge request earns its own level of autonomy from evidence, and loses it exactly where production proves it wrong.

Comment

Check every claim yourself, one link each: CLAIMS.md

Inspiration

Small teams don't let automation ship to production on its own, and they're right not to: nothing tells them which automated decisions are safe. Pipelines are either always manual or always automatic. We wanted autonomy to work like trust between people: earned per change, backed by evidence, lost where it fails, and earned back slowly.

What it does

Every merge request on the 46k-star FastAPI full-stack template (backend, React frontend and Terraform-managed Cloud Run) gets evidence, a risk score and an autonomy level:

  • Hands-off: merges and ships to production with no human. A scheduled job merges only a current Hands-off verdict from the Router agent's own account, and never changes to the system's own controls.
  • Supervised: one human approval. Assisted: a human drives each step. Blocked: doesn't ship.

The evidence comes from:

  • tests (58 backend, 62 Playwright UI);
  • a replay of 166 recorded real requests;
  • static reachability from HTTP routes to each vulnerable function, with runtime hit counts;
  • a Terraform plan read at attribute level.

Five GitLab Duo agents add judgement the rules can't encode, and they can only ever raise risk:

  • the Planning agent scores an issue's blast radius before code exists;
  • Triage gives a reachability verdict per advisory and opens fix MRs;
  • the Infra agent reads the plan;
  • the Autonomy Router reads the diff and posts the proof card;
  • Rollback opens incidents.

After every release, a production monitor replays canary traffic against the new and previous Cloud Run revisions on the same prod data, and rolls back automatically on any difference. Each outcome goes into a trust ledger in GCS, which recalibrates the thresholds and adds or halves per-area penalties. All of it is public on a ledger page.

Highlights (all live, all linked from the README and CLAIMS.md)

  • An agent's fix reached production with no human step: a vulnerable certifi landed on main; the Triage agent opened the fix MR (!52) at 03:32, the scheduled job re-ran its checks as a person (GitLab starts no Duo flow on a bot's pipeline), the Router rated it Hands-off, and it was merged at 03:54, promoted at 03:59 and confirmed clean by the monitor at 04:01. 35 minutes from advisory to fixed prod.
  • Zero clicks from MR to prod, for a person's change too: MR !31 was opened at 19:42, rated Hands-off at 19:47, merged by the system at 19:54, promoted at 19:58, and confirmed clean by the monitor at 20:02.
  • Caught in prod, rolled back in 3.3 s: a person shipped a change rated Assisted; the monitor saw 50 items instead of 100 and pinned traffic back.
  • Trust moves both ways: the next clean change to that route was held at Supervised for exactly that reason. Clean releases halved the penalty, and the Hands-off threshold widened 0.20 → 0.25 → 0.30 on its own.
  • Agents caught what rules missed: the Infra agent flagged a prod timeout cut that would also have stripped the rollback tags. Reviewing the auto-merge code itself, the Router found three security flaws, all fixed before merge.
  • A pre-registered benchmark: 12 seeded changes, with expected levels committed before any run. It scored 10/12 as run, with all safe changes Hands-off and both misses on the lenient side. We published the misses, fixed them (an infrastructure ceiling; access widening now blocks), and reached 12/12. A live run on Oct 27 found one more (a change adding a known advisory came out Hands-off at exactly the widened threshold); it was published and fixed within the hour: adding an advisory is now at least Supervised.
  • Not tied to one app: the same pipeline and agents run on VaultScope, an unrelated FastAPI app with 350 tests, producing Hands-off, Supervised (with the planning prior applied) and Blocked verdicts.
  • Sustainability: evidence costs only what it can tell you. A documentation-only MR now skips the tests, replay and browser tests it can't change: 7.3 → 0.77 runner-minutes (−89%, about 0.5 g CO2e each), measured on our own MRs; that's about 6% of all MR CI on the template's real PR history. Plus a carbon delta on every infra change, a CI footprint per release on the ledger page, and prod scales to zero.

How we built it

  • GitLab: CI across the DevSecOps stages, and Duo Agent Platform custom flows triggered by pipeline events, work items and failed pipelines.
  • Google Cloud: Cloud Run, Artifact Registry, Secret Manager, GCS (state and ledger) and Workload Identity Federation from GitLab OIDC. No service-account keys exist; the merge token can only be read by pipelines on main.
  • Our tools are Python, in tools/: a call graph with FastAPI route discovery, traffic record and replay with ID mapping, the risk engine and ledger, Terraform plan analysis, the proof card, and the monitor.

Challenges we ran into

  • GitLab starts no flow for bot-initiated pipelines, so agent-opened MRs never got a Router review. The scheduled merge job now re-runs such a pipeline once, as a person; the merge rules don't change.
  • Duo's HumanInput step paused but never resumed in our tests, so human gates use MR approvals and manual jobs.
  • Developers can't set CI variables in this setup. The merge token moved to Secret Manager behind keyless federation, which turned out to be more secure.
  • Real failures we hit and fixed, all in the history:
    • a frontend build that baked in localhost:8000, caught by the UI tests before prod;
    • a missing workspace manifest that failed an image build after merge, so images are now built on MRs;
    • a flaky API read that routed a Hands-off release to a manual gate, fixed with retries;
    • a Terraform lock race, fixed with lock-free MR plans.

Accomplishments that we're proud of

The system has a memory. It held a perfectly clean change because the last change to that route failed in production. It later widened its own autonomy after a run of clean releases. And when we measured it, we published the misses.

What we learned

Trust has to be scoped, and it kept teaching us that: twice, trust earned on small changes widened one global threshold enough to let a riskier kind of change through (infrastructure in the benchmark, a new advisory live). Each kind of risk now has its own ceiling. Evidence also beats severity: Jinja2's 3 advisories were 0 reachable on VaultScope.

What's next

  • A Validation agent that classifies replay diffs as intended or regressions.
  • Trust ceilings for more kinds of change, beyond infrastructure and new advisories.
  • Packaging the pipeline as an includable CI template.

Built With

Share this project:

Updates

Submission history