Earned Autonomy

A delivery agent that has to keep score before it may merge on its own.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for Earned Autonomy

Inspiration

Every team that tries agents in its pipeline hits the same question. The agent can write the change, but should it be allowed to merge it? Today the answer is a gut feeling, or a confidence the model reports about itself that nobody ever checks.

Weather forecasters solved this a long time ago. Nobody trusts a forecaster because it sounds sure. You keep its forecasts, you wait, you score them, and the Brier score tells you whether "30% chance of rain" really means rain on three days out of ten. I wanted a delivery agent held to the same standard.

What it does

When an issue is assigned to the flow's service account, Earned Autonomy settles the outcomes of the merge requests it forecast earlier, implements the issue with tests (create), opens a merge request and writes down one number before the pipeline runs, the probability that this change breaks main once merged. That number goes into an append-only ledger in the repository.

Once the merge has gone through the main pipeline and the post-deploy smoke check on GitLab Pages, the outcome is appended next to the forecast. The pair is scored. From the score, deterministic code that the agent cannot edit computes its level.

  • assisted, at the start. Draft merge request, a human at every step.
  • supervised, after 10 resolved forecasts with positive Brier skill. The agent runs everything, a human approves, a second flow merges on approval.
  • hands-off, after 20 with a skill above 0.10. The agent merges on its own when the calibrated risk is 15% or less.

A failure it called at 10% or less is a surprise, and each recent surprise costs a level at once. The track record, the reliability diagram and every forecast with its evidence are public on the project's Pages site, rebuilt on every merge.

How we built it

  • Two custom flows on the GitLab Duo Agent Platform. The delivery flow has three agents in sequence, settle, implement, forecast and gate, triggered by Assign and Mention (plan). The merge flow is triggered by the Merge request Approved event and refuses to merge anything that was not forecast first.
  • A small Python package, standard library only. Brier score, Brier skill against the base rate, Platt recalibration fitted by damped Newton with a prior on the identity, the level rules and the gate. 178 tests.
  • A pipeline that reaches every stage, tests and the ledger guard (verify), SAST and secret detection (secure), a wheel published to the project's package registry on every main pipeline (package), Pages deployment to a production environment (release, configure), a smoke job that waits for Pages to serve the new commit and checks the page (monitor), and an append-only ledger with a policy file only humans may change (govern).
  • Built by one person directing coding agents, every change went through merge request review.

Challenges

One design problem showed up before any run. If the agent never breaks main, the base rate is zero, the constant "never fails" is perfect, and no agent can beat it, so a perfect record would earn nothing. The reference is pulled toward a 10% prior.

The second one showed up after six clean merges. The first scoring run used the smoothed rate known at the end of the ledger, a number nobody could have known when the forecast was made. Six honest forecasts of 4% to 12% ended at a skill of -0.87, and the agent would never have left assisted. The reference now follows the rate known at each forecast. The agent still has to say lower than the reference to gain anything, so a cautious 12% on a docs change costs it, and that is the point.

The rest was plumbing. The flow sandbox runs Python 3.9 without pip, so the package supports 3.9 and the CI tests both 3.9 and 3.12. The flow validator refused some fields from the published spec. Pages on new hackathon projects were members-only at first, so the smoke check got a 403 until GitLab opened them for every project. And one carriage return written on Windows broke the package version in the registry.

Accomplishments

Figures as of 9 October 2026.

  • 36 merge requests from the agent, merged after review, and main never broke. 35 forecasts resolved, every one held.
  • The agent is still at assisted. Its Brier skill went from +0.84 after its first forecast down to -0.53, and is back at -0.22. Its forecasts of 12% and 15% sat above a base rate that was already near 5%, and the score keeps that. It earns supervised the day it beats the base rate, not before. The rule was left alone rather than tuned until the badge turned green.
  • After ten resolved forecasts, Platt recalibration switched on by itself and now pulls the agent's raw 3% to 8% down to well under 1% at the low end and about 2% at the high end.
  • It pushes back. Asked for a license classifier that setuptools 77 rejects next to an SPDX license expression, it left it out and said why. Asked to edit the README under ledger/, it refused because that folder is append-only, and proposed docs/ instead. Told so, it did it there. And it takes review. Its README once said the guard blocks any change to the protected files, which is wrong, a human may change them. One comment on the merge request, and it rewrote the sentence, added a test for it and pushed again. Told it could not touch score.py, it read the prior from the function signature rather than break the rule, and a human moved the constant afterwards.
  • The guard covers the policy, the scoring code, the smoke check and the pipeline, and it runs from the target branch. A merge request cannot switch off the check that judges it.

What we learned

  • The first reference used information from the future, and it made an honest agent look hopeless.
  • Guarding a file is not enough. The guard has to run from code the merge request cannot touch, or the agent can rewrite the referee in the same change.
  • Test the guard against the real author, not a made up one. The guard looked for an ai- name, the flow commits as Earned Autonomy from a service_account_ address, and every test was green. Replayed on a real agent branch, the old guard let a change to the scoring code through. It now reads the email.
  • A calibrated agent on a clean codebase should sound almost bored, 2% and 3%. The drama is in the one merge where it says 30% and is right.

What's next

  • Score more than one event, for example time to recover, not just failure.
  • Let several agents share one ledger per project, so the one with the better record gets the riskier work.

Links

Built With

  • gitlab
  • gitlab-ci
  • gitlab-duo
  • gitlab-pages
  • python
Share this project:

Updates

Submission history