Stripes: AI agents earn their autonomy

Every agent merge request makes a prediction. A Cloud Run canary checks it on real traffic. Kept bets earn the agent more freedom, broken ones roll back and demote it, and only a human can promote.

Comment

Inspiration

AI agents can now write code, open merge requests and ship. Every team then hits the same question: how much do we let them do on their own? Today the answer is all or nothing. Either a person reviews every change, which throws away most of the value, or the agent ships with no check, which nobody trusts.

We don't treat people that way. A new engineer starts with small tasks, gets reviewed, and earns more freedom by being right. We wanted the same thing for agents, based on evidence from production instead of a gut feeling.

What it does

Stripes gives each type of change (docs, dependencies, features) an autonomy level:

  • assisted: an owner merges and deploys
  • supervised: the pipeline runs, but an owner clicks before production
  • hands-off: issue in, live in production, nobody touches anything

Levels are earned, never assumed:

  1. Every agent merge request makes a prediction. For example: "p95 latency on /search stays under 250 ms" with a stated confidence. A linter rejects merge requests without a valid prediction block.
  2. A canary settles the bet. Stripes deploys the change to a 10% Cloud Run canary, sends real load, and a statistical judge compares canary to stable with bootstrap intervals. Each prediction is kept, broken or inconclusive.
  3. A ledger turns outcomes into trust. Kept predictions build a reliability lower bound (Beta distribution) and a calibration score (Brier). An agent that is confident and wrong loses trust faster than one that is honest about doubt.
  4. A broken prediction costs the agent. Stripes rolls back in seconds, demotes that change type, opens an incident with the evidence, starts a fix agent, and starts a postmortem agent that proposes a new review rule.
  5. Only a human can raise a level. When the record is good enough, Stripes opens a promotion merge request with the evidence. A policy guard in CI refuses any level raise that did not come through an owner's merge.
  6. A kill switch freezes all deploys and revokes the agent's token in one job.

How we built it

  • GitLab Duo Agent Platform: two custom flows (build and postmortem) triggered by assigning an issue to a service account, the foundational developer flow for incident fixes, plus AGENTS.md, a prediction skill, subagents and custom review instructions.
  • GitLab CI/CD: classify, test, build, secure, plan, deploy, verify, release, record and handoff stages. It uses dynamic child pipelines generated per change, resource groups so only one canary runs at a time, protected variables, manual gates for supervised changes, and scheduled pipelines that run the backlog, refresh baselines and build the dashboard.
  • GitLab security: SAST, secret detection, dependency and container scanning as a gate. A change cannot ship if it adds a finding.
  • Google Cloud: Cloud Run with revision tags and traffic splitting for the canary, Artifact Registry, Firestore for app data, Cloud Storage for the public dashboard, and Workload Identity Federation, so CI holds no service account keys.
  • Stripes engine: Python 3.12, pydantic v2, strict type checking, 462 tests including Hypothesis property tests that prove a level can never rise without a verified owner promotion.

Proof it runs

  • The agent took a docs issue to production with no human step: issue, agent merge request with predictions, bot merge, canary, verdict, promote, release, issue closed.
  • In a game-day drill, an agent change claimed search would stay under 250 ms. The canary measured 527 ms over 1,482 samples. Stripes rolled back in 3.0 s, demoted features to supervised, opened incident #25, and started the fix and postmortem agents.
  • Features then earned three clean changes, and Stripes opened a promotion merge request that the owner merged on camera.
  • Today: docs at hands-off with 17 kept and 0 broken, features back at hands-off with 17 kept and 1 broken. The project has 879 pipelines and 44 merged merge requests.

Challenges

  • Protected secrets vs agent branches. Agent branches cannot see protected variables, so a main-side pipeline plans each merge request through the job-token trigger API.
  • Honest statistics on small samples. A demo profile lowers the evidence bars so the loop shows within days, while the production profile keeps strict ones.
  • Races between agents. Two promotions could open at once after a merge. The ledger now only restores a level after a real demotion, and commits retry on a stale ledger.
  • Staying at zero cost. Cloud Run scales to zero, canaries are retired after each verdict, and a cost check runs against free-tier limits.

What we learned

Trust in an agent is not one number. It is per type of change, it decays when the agent overstates its confidence, and it has to be earned back through a person. Making the agent state a measurable prediction turned out to be the key. Once a bet is written down, production can settle it.

What's next

Package Stripes as a CI/CD catalog component, so any GitLab project can include it with a few lines, and add more prediction kinds such as error budgets and cost per request.

Built With

  • fastapi
  • firestore
  • gitlab
  • gitlab-ci
  • gitlab-duo-agent-platform
  • google-artifact-registry
  • google-cloud
  • google-cloud-run
  • hypothesis
  • jinja
  • opentelemetry
  • pydantic
  • pytest
  • python
  • scipy
  • workload-identity-federation
Share this project:

Updates

Submission history