tenPercent

Six changes from issue to production with zero human steps. tenPercent: GitLab Duo agents that ship through a 10% canary, roll back on real telemetry and stop when the error budget says so.

Comment

Inspiration

Small teams ship fast, and nobody watches production at 3am. A deploy that slows down payments, or quietly fails one payment in fifty, can run for hours before a customer complains. In card payments that means abandoned checkouts, double charges and lost revenue. The usual answer is an on-call engineer and a release process full of human approvals, which a four-person team doesn't have.

AI coding agents make this gap wider. They can open merge requests all night, but someone still has to decide whether a change is safe for customers, notice when it isn't, and undo it. At the same time, security problems are found faster than teams can fix them: Verizon's 2026 Data Breach Investigations Report puts the median time to fix a vulnerability at 43 days.

So we built the missing half. tenPercent isn't another agent that writes code. It's the night shift: it takes a request from one person, ships it, proves it on real production traffic, and takes freedom away from the agents when production starts to suffer.

The name comes from the core idea: every change has to prove itself on ten percent of real traffic before it reaches everyone.

What it does

One person opens a GitLab issue. Everything after that happens without a human.

  1. Plan. A team of GitLab Duo agents starts on the new issue. A triage agent turns the request into acceptance criteria and a test plan, and decides whether it's a feature or a fix, and how risky it is.
  2. Build and review. A builder agent writes the code and tests and opens a merge request. A separate reviewer agent checks it critically and runs the tests itself. A security agent goes through the security scan results and fixes any real problems in the new code.
  3. Merge safely. GitLab doesn't let AI agents merge code, and we kept it that way. The agents only mark the merge request as ready. A fixed rule in the CI pipeline merges it, and only if the review and security checks were done, nothing is unresolved, every job passed, and the agents didn't touch any of the files that define their own limits.
  4. Prove it. The pipeline:
    • builds the app;
    • runs six security scans and checks that every library is on an approved list;
    • updates the cloud setup with Terraform;
    • deploys to a staging copy;
    • sends 10% of real production traffic to the new version on Google Cloud Run.
  5. Let production decide. The app sends live telemetry (traces, metrics and logs) to GitLab Observability. A judge compares the new version with the current one. The new version stays only if:
    • it handled enough requests to judge fairly;
    • almost none of them failed;
    • payments stayed fast (under half a second for 95% of them);
    • it wasn't noticeably slower than the current version.
  6. Release or roll back. If it passes, the exact version that was tested goes to 100% of users. A GitLab Release with an evidence file is published, and the agent posts a "Shipped" summary on the issue. If it fails, all traffic goes back to the previous version automatically.
  7. Fix itself. After a rollback, a separate fix team of agents takes over:
    • it opens an incident;
    • it explains the root cause using the slowest parts of the failed requests;
    • it writes a fix, has it reviewed, and ships it through a second 10% test.

If there's no safe fix, it stops and asks a person to decide, instead of shipping something that would fail again.

  1. Earn autonomy. tenPercent keeps an "error budget": a small allowance of slow or failed payments per week. While there's budget left, agents ship features and fixes on their own. When the budget runs out, the system freezes new features by itself and only lets fixes through, until production recovers.
  2. Stop it anytime. Adding one label to an issue stops every agent at its next step.
  3. Ask it questions. Anyone can mention the on-call agent in an issue and get an answer backed by evidence ("why was this rolled back?", "can we ship features right now?"). A read-only assistant answers the same questions in GitLab Duo Chat.

The app being shipped is a small card payments API (no real money). It handles payments in whole cents, can't charge twice for the same request, enforces a daily spending limit, blocks accidental double charges and emails receipts. A public website lets anyone make test payments and see which version answered.

How we built it

GitLab Duo Agent Platform

  • Three agent flows, all stored in the repository and published to the AI Catalog from it:
    • the Conductor: six agents that take an issue all the way to a release;
    • the fix flow: three agents that respond to a rollback;
    • the on-call agent: answers questions.
  • One custom agent, the read-only Release Analyst, for GitLab Duo Chat.
  • Triggers: opening an issue starts the Conductor; a mention starts the fix flow and the on-call agent.
  • Each flow runs under its own account, so every commit and comment shows which agent did it and for whom.
  • AGENTS.md is the rulebook every agent follows: money in whole cents, never charge twice, protect personal data, never weaken a test, never change the safety limits.

GitLab CI/CD runs the deterministic part: build, scan, deploy, test on real traffic, judge, and release or roll back. Agents decide; the pipeline proves.

GitLab Observability stores the live telemetry. The judge reads it through GitLab Observability's MCP connection.

Google Cloud

  • Cloud Run hosts the app and splits traffic 90/10 between versions.
  • Firestore stores payments; Artifact Registry stores the app images.
  • Everything is set up with Terraform.
  • The pipeline logs in to Google Cloud without any stored passwords or keys.

The app is written in Python with FastAPI and has over 175 automated tests.

Built to the GitLab Security Standard. Each of its five layers is enforced by the pipeline, not just written in instructions:

Layer How tenPercent does it
Authorize Agents can't merge, and can't change the files that set their own limits. A named person owns those files
Isolate Agents work in a sandbox with restricted internet access, and never get production credentials
Verify Six security scans, plus a check that every library is on an approved list
Release Production runs exactly the image that was scanned and tested, checked by its fingerprint before release
Respond A stop label, traces of every slow request, automatic rollback, incidents and the fix flow

All nine DevSecOps stages are covered: Plan, Create, Verify, Package, Secure, Configure, Release, Monitor and Govern, each by a real agent or pipeline job.

Challenges we ran into

We hit real platform limits and documented every one in the README, without cutting corners on "hands-off":

  • AI agents can't merge code in GitLab. We kept that safety rule. Agents mark a change as ready, and a fixed pipeline rule does the merge.
  • Agent sessions end after about an hour, and one flow can't start another. We measured session length, tried five different ways to start the fix flow, and kept the one that works: the rollback job hands over to the fix flow with a mention.
  • A test that passed didn't hold in real life. One trigger worked when we tested it ourselves, but the first real rollback showed GitLab ignores it when an agent caused the change. We found that out from the failure and replaced it.
  • Most of our telemetry was missing at first. About 85% of traces never arrived because of how Cloud Run marks requests. We tracked it down and now capture almost every request.
  • We were measuring the wrong thing. Early speed numbers included network time between GitLab and Google Cloud. The judge now uses the app's own timing.
  • Releases weren't shipping the exact tested image. We changed release so it deploys the image's exact fingerprint and refuses to continue if anything differs.

Accomplishments that we're proud of

  • Six changes went from a person opening an issue to production with no other human step: request IDs, receipt emails, a daily spending limit, security headers, double-charge protection and a privacy fix. The median time from issue to production is 28 minutes.
  • Security fixes in minutes, not weeks. Our own security scan found missing browser protections; the agents fixed them in production 26 minutes after the issue was opened. A privacy leak (full cardholder names visible to anyone) was fixed in 29 minutes.
  • The agents caught their own bug. For the daily spending limit, they spotted on their own that two payments arriving at the same moment could slip past the limit, fixed it, and proved it with a test that fires 20 payments at once.
  • A real failure ran the whole safety loop. One change made payments too slow in production (0.68 seconds against a 0.5 second target). It was rolled back automatically. The fix team opened an incident, found the exact cause and proposed a fix. Then its own reviewer refused that fix, because it wouldn't have solved the problem for new customers, and the only faster option could have charged unknown accounts. It asked a person to decide, with three clear options.
  • The system limited itself. That failure used up the error budget, so tenPercent froze new features on its own, while a privacy fix still went all the way to production with no human step.
  • Agents never took a shortcut. They never changed a safety limit, the test traffic or a test to make a check pass.
  • Everything is public and checkable: every pipeline, release, agent comment and incident, plus a live website showing real numbers.

What we learned

  • Autonomy should be earned, not switched on. The best control over agents wasn't a smarter prompt. It was a budget based on what production actually experienced.
  • Agents are best at judgement, pipelines at proof. Agents understand requests, review code and explain failures. The pipeline moves, measures and decides by fixed rules.
  • A rule in a prompt is a wish; a rule in the pipeline is a guarantee. Every rule we cared about ended up as an automatic check.
  • Test in real conditions. Our tests and our trigger experiment both passed in conditions that weren't real. Production showed the truth.
  • The best agent knows when to stop. The strongest moment of the week was an agent refusing its own team's fix and asking a human.

What's next for tenPercent

  • Verify new accounts when they sign up instead of at payment time (the decision a person made after the failed change), as a new issue for the agents.
  • Judge on more signals: per-feature budgets, cost per request and payment approval rates.
  • Sign every release so anyone can verify that production runs exactly what was tested.
  • Package the merge rule, the judge, the library check and the error budget so any GitLab project can give its agents a night shift.

Built With

  • fastapi
  • gitlab
  • gitlab-ci
  • gitlab-duo
  • gitlab-duo-agent-platform
  • gitlab-observability
  • gitlab-releases
  • gitlab-security-scanning
  • gitlab-terraform-state
  • google-artifact-registry
  • google-cloud
  • google-cloud-firestore
  • google-cloud-run
  • javascript
  • model-context-protocol
  • opentelemetry
  • opentofu
  • pydantic
  • pytest
  • python
  • terraform
  • workload-identity-federation
Share this project:

Updates

Submission history