MRDoc decides how much freedom an AI coding agent gets for each merge request on GitLab: Assisted, Supervised or Hands-off. The level comes from a risk score of the diff and from the trust that part of the code has earned in canary releases on Google Cloud Run.
Inspiration
Today a coding agent gets one setting for a whole project: either it ships on its own, or a person checks every step. Both are wrong most of the time. A new line in the FAQ and a change to how checkout calculates tax are not the same risk. And an agent that has shipped many good changes to one part of the code has earned more freedom there than an agent that broke it yesterday.
So I asked: what if an agent's autonomy worked like trust between people? You earn it with good work, you earn it per area, and you lose it fast when something breaks.
What it does
Every merge request an agent opens gets one of three levels:
- Assisted: a person approves the merge request, the canary (a test release on part of the traffic) and the full release.
- Supervised: the merge request merges by itself and the canary starts by itself. A person clicks once to go live.
- Hands-off: nobody needs to act. Merge, canary and full release are automatic.
The level comes from two questions:
- How risky is this change? MRDoc scores the diff and the repository. It never trusts the agent's own description of its change.
- How much has this part of the code earned? Each module (content, catalog, checkout, ...) has a trust ledger, and MRDoc changes it only from real production results.
GitLab enforces the level, not a prompt. The level decides which of the agent flow's privileges the Flows API pre-approves, whether a person must approve the merge request, whether it merges by itself, and whether the release jobs are manual or automatic.
Every change is tested in production first. It goes out as a canary on Cloud Run: 10% of real traffic, plus synthetic test requests. If the canary passes, the module earns trust after a probation window. If it fails, MRDoc rolls the change back automatically, whatever its level, opens an incident, lowers the module's trust, and starts an incident-fix agent flow at the level the module has after the failure. When the fix passes its own canary, a postmortem flow writes the postmortem and closes the incident.
A public dashboard shows each module's trust, the level a typical change would get now, and the history of every decision and release.
Live demo
The demo ran end to end on GitLab.com and Google Cloud Run, with real agent flows:
- content, Hands-off (!10). An agent changed the shop banner. Nobody touched it. The canary sent 167 requests to each version, saw 0 errors and no broken business rule, and the change went live by itself.
- catalog, Supervised (!11). An agent added a
Cache-Controlheader. The merge request merged by itself, and the release waited for one click. - checkout, Assisted (!9). An issue asked for discount codes to apply after tax. Checkout is a critical module with little trust, so a person had to approve the change.
- 4,011 tests passed. The demo is set up this way on purpose: the shop's unit tests check discounts and tax separately, never together, and the agent's new tests expected its own after-tax result.
- In the canary every request succeeded, so the error test saw nothing. But the invariant "discounts apply before tax" failed on 64 of 83 checked quotes.
- MRDoc rolled the change back, opened incident #12 and cut checkout's trust at its typical risk from 0.050 to 0.013.
incident-fixstarted at Assisted, wrote a plan and waited for approval. Its fix (!12) came with a regression test that fails first and passed its own canary. Then thepostmortemflow closed the incident.
Earlier agent changes built the trust you see in the demo: content went from Assisted to Supervised to Hands-off over real releases. Nothing was written into the trust ledger by hand. The demo used the policy's demo profile, with a 5-minute probation instead of 24 hours, so trust could move within one session.
How MRDoc decides
Risk borrows from FMEA, the engineering method that rates a failure by how bad it is, how likely it is and how hard it is to notice. Severity \(S\) is the blast radius: the tier of the most critical module that owns or imports the touched files, plus 0.2 when the change removes or changes part of the public API (or that check's report is missing). Occurrence \(O\) comes from the size of the change, how spread out it is, how often the touched files changed and needed fixes in the last 90 days, new complexity and dependency changes. Undetectability \(D\) is high when tests do not cover the change or the canary cannot see it. All three are between 0 and 1:
$$ r = (S \cdot O \cdot D)^{1/3} $$
Six hard gates come first, and a low score cannot outweigh them. Four force Assisted: a change to a governance file, a new critical or high security finding, too much risk in one module within 24 hours (against splitting a risky change into small ones), and a bug fix without a failing-first regression test. Two cap the level at Supervised: irreversible paths (migrations, the payment integration) and critical modules.
Trust is a cautious lower bound, not an average. Every result in the ledger has a weight \(w_i = 0.97^{a_i} \cdot m_i\): it loses 3% per day of age \(a_i\), and \(m_i = 0.5\) for evidence from an older agent model (1 otherwise). For a change of risk \(r\), MRDoc counts two things. \(W(r)\) adds up the passed canaries of changes that were at most 0.05 less risky, each also weighted by the canary's statistical power \(p_i\). \(F\) adds up the failures of any risk: failed canaries, and reverts, hotfixes, incidents or SLO breaches found later:
$$ W(r) = \sum_{\text{passes } i \text{ with } r_i \ge r - 0.05} p_i \, w_i $$
$$ F = \sum_{\text{failures } j} w_j $$
$$ \text{trust}(r) = \text{the } 0.05 \text{ quantile of } B(r) $$
$$ B(r) = \mathrm{Beta}\big(1 + W(r),\; 1 + 3F\big) $$
In plain words: MRDoc is 95% sure that the module's real pass rate for changes this risky is at least this high. A pass vouches only for changes no riskier than itself (with a 0.05 margin), so many tiny changes cannot buy trust for a risky one. A failure counts at once, for changes of every risk, and three times as much as a pass with full canary power.
The level. The trust a change needs grows with its risk, so a module is trusted with small changes long before big ones:
$$ \tau_{\text{Supervised}}(r) = 0.10 + 0.60\,r $$
$$ \tau_{\text{Hands-off}}(r) = 0.30 + 0.65\,r $$
A module gets Hands-off when its trust at risk \(r\) reaches the Hands-off line, Supervised when it reaches the Supervised line, and Assisted otherwise. A change gets the lowest level of the modules it affects, directly or through the code that imports it, and a gate can lower it further.
The canary verdict. The synthetic traffic is sized by a standard power calculation (normal approximation): 167 requests to each version give about an 80% chance to catch a rise in the error rate from 1% to 6%. The canary fails on a one-sided Fisher exact test of the errors, which is stricter (about 74% at that size), on a p95 latency more than 1.5 times the stable version's, or on any of nine business invariants. The invariants catch answers that succeed but carry wrong data.
How I built it
GitLab Duo Agent Platform. MRDoc starts the foundational flows through the Flows API: developer/v1 for each labelled issue and fix_pipeline/v1 for a failed pipeline, both with the pre-approved privileges of the decided level, and code_review/v1 on every agent merge request. Four custom flows are written for the AI Catalog: incident-fix and postmortem run in the loop, and explain-decision and risk-review are optional (risk-review stays off until its behaviour is confirmed). The flows ran on GitLab Duo's default model, Claude Sonnet 5.5. AGENTS.md gives agents the module boundaries and the rules, for example "never edit governance paths" and "every bug fix needs a failing-first regression test". A validator checks the flows and the agent config in CI, because one invalid config file blocks every flow start.
GitLab CI/CD. Merge request pipelines run agent-written code without any secrets. Only the control plane and the default-branch pipeline, with protected variables, make decisions. The default-branch pipeline generates a child pipeline for the decided level, with manual or automatic jobs. SAST, Secret Detection and Dependency Scanning feed one of the hard gates. The trust ledger lives on its own branch, so every change of trust is a commit: a free audit log.
Google Cloud. Cloud Run runs the demo shop and the control plane. Canaries use Cloud Run revision tags and traffic splits. Artifact Registry stores the images, Secret Manager the tokens, and Cloud Monitoring provides request counts per revision to catch late failures. Every component runs as its own service account with minimal roles, and one idempotent script sets it all up.
Code. Python 3.12. engine/ is a deterministic library (risk, trust, decision, canary statistics) with no network access. controlplane/ is a FastAPI service that owns the ledger, runs the engine and talks to GitLab. Every tunable number lives in one policy file. There are 4,026 tests (4,021 pass; 5 need shellcheck and are skipped), including property-based tests of the design's promises: risk never drops when a factor grows, gates always win over the score, and the level never drops when trust rises.
The repository itself was written by coding agents (Claude Code) working from a written brief. Every choice the brief left open, or the platform forced, is recorded in docs/ASSUMPTIONS.md (71 decisions).
Challenges I ran into
- Custom flows cannot ask for approval per tool. The flow schema rejects per-tool approval, and a custom flow started through the API ignores the privileges you send. So Assisted inside
incident-fixandpostmortemis an approval step before the first write. The docs do not say whether such a step pauses a session started through the API. A live test showed that it does. - Write tools could not be pre-approved. Started from a workflow definition, a Developer Flow kept its write privileges, but GitLab pre-approved only the read-only ones. Starting Supervised and Hands-off runs through the flow's AI Catalog consumer got all five privileges pre-approved.
- GitLab shows the token owner as the author. Flows act with a composite identity, so every agent merge request looks like it was opened by the person behind the token. MRDoc recognises agent work from the commit author, the
duo/branch prefix and author patterns in the policy. - Agent code runs in CI. Coverage, the API diff and the regression check come from pipelines that run untrusted code. A missing or invalid report counts as the worst case, and the control plane recomputes everything it can from the repository itself.
- Silent failures. A change that returns correct-looking answers with wrong data passes unit tests and a plain error-rate test. The business invariants are a governance path, so an agent's merge request that touches them always needs a human's approval.
- Limited permissions in the hackathon workspace. The design also requires
CODEOWNERSapproval, a protected production environment and branches that only the control plane can push to, but the workspace's Developer role could not turn them on. Until a Maintainer does, any Developer, the agents' service accounts included, could push to the default branch or thetrust-ledgerbranch directly. The README lists this as a residual risk.
Accomplishments that I'm proud of
- The full loop ran live: all three levels, a planted bug that only the canary caught, a real rollback and a real loss of trust, shown in a demo video of under three minutes (the waits are cut).
- Through merge requests, agents cannot change the rules they are judged by on their own: touching a governance file forces Assisted and needs a human approval.
What I learned
- Test coverage shows that code ran, not that anything was checked. Only production, through the canary, the invariants and the probation window, shows whether a change was good, so trust is earned there.
- A canary only sees what it is built to see. It misses small rises in errors, and wrong answers that no invariant describes.
- Enforcement must sit in several layers, because the platform's approval features do not cover every flow type.
- The weights and thresholds are policy choices, not measurements. That is why they live in one file, and an agent's merge request that changes it always needs a human's approval.
What's next for MRDoc
- Fit the risk weights and trust thresholds to real production outcomes.
- Keyless CI with Workload Identity Federation instead of a service account key.
- Per-tool approvals inside custom flows, if GitLab adds them to the flow schema.
- Run more than one control plane instance, and reconcile webhook deliveries that GitLab never retries.
Log in or sign up for Devpost to join the conversation.