Pocketful Autopilot

AI writes the code and the tests. Autopilot tests the tests: one GitLab issue in, production out, with mutation and security gates, automatic rollback and a receipt that counts human actions.

Comment

Pocketful Autopilot is for small teams that ship a payments API with no QA, security or SRE person. AI writes their code and their tests, so who tests the tests? Our imported app had 99.70% line coverage, yet mutation testing found 13 changes that no test noticed. A test route returned every password hash. We left it open on purpose, as a job for the agents. With Autopilot, one GitLab issue starts a Duo flow, and agents fix what the CI gates find, including a mutation gate. In our hero run the change reached production with 1 human action and 9 of 9 stages, and the receipt came at 28m24s.

Path B · Hands-off · Cloud Run + Workload Identity Federation · 9 of 9 stages

Verify it in 5 minutes (no sign-in needed):

  • Hero receipt on issue #3: "Human actions: 1", Approvals 0, Manual jobs 0, 9 of 9 stages with links.
  • MR !2: three agent commits, one per round, merged by the flow's service account.
  • Hero main pipeline 2962311393: all green, no manual job (monitor:rollback skipped: the watch passed).
  • Game-day main pipeline 2971404006: red on purpose. monitor:watch failed (p95 1569 ms against 800 ms), then monitor:rollback passed.
  • Incident #7: opened by the flow. The owner closed it about five hours later, which lets runs start again.
  • Live /health: with curl -si, the x-app-version header is afecd876..., the commit the flow merged.
  • README: the run diagram, the gates and "Honest limits".

Before you check, five things that can look wrong:

  • Names. MR !2 shows the owner as its author, and the flow's commits show his name. This is composite identity: a Duo flow acts for the person who started it. The commits' email and the merge belong to the service account. Until the receipt, issue #3 shows the owner only at 08:38:48, the second it was filed.
  • Environments page. Production's last deployment is monitor:rollback on 222922ee: the game-day rollback, which restored the hero release (/health answers afecd876). The rollback and incident in the hero receipt's Monitor row ran only on game day.
  • Staging keeps its /_test routes on, on purpose, with throwaway data. Production has them off (404).
  • "Skipped" pipelines are the owner's pushes with ci.skip: to main (docs, setup and fixes between runs) and the tag submitted-2026-10-23. The Duo session page needs a sign-in.
  • Two times for one deployment. The receipt's event list shows each deployment at the time its job started (staging T+17m29s, production T+22m52s). Its Timeline uses the time the job ended (T+18m37s, T+23m48s). This page uses the Timeline.

Inspiration

Pocketful is a small payments API. An AI agent team built it in an earlier hackathon: 248 tests, all green. Then we asked who checks the checks. Mutation testing changed the code in 180 small ways in two files, and 13 of those changes survived. A test-only route, GET /_test/export, needed no login and returned every user's email and password hash.

That route breaks a house rule that we wrote for this project: test-only routes are off unless POCKETFUL_TEST_HOOKS=1 (AGENTS.md section 5, in the repository since 2026-10-13). The imported app does not follow it. We left it that way on purpose, as a job for the agents. The team writes the rule once. The gates enforce it. The agent applies it.

A green pipeline only proves that tests ran. We wanted the tests checked inside GitLab, with no human in the middle, and with a receipt that proves what happened.

Mutation testing is not new: Stryker and PIT exist, and Google shows surviving mutants in code review. Mutation-guided LLM test generation exists too: Meta's ACH (arXiv 2501.12862) and the MuTAP research method (arXiv 2308.16557). What we add is where the mutant sits. A surviving mutant is a failing merge gate inside a hands-off GitLab Duo flow. It shows up as a failing test in the merge request, and the agents must beat it before anything ships. (Google: Petrović and Ivanković, "State of Mutation Testing at Google", ICSE-SEIP 2018; and Petrović, Ivanković, Fraser and Just, "Practical Mutation Testing at Scale", arXiv 2102.11378.)

What it does

One person files one issue. GitLab does the rest, then proves it. Autopilot tests the tests in three layers:

  1. Criteria are frozen at the first push. A planner agent writes 3 to 8 numbered acceptance criteria. The guardrails job fails any later change to them.
  2. Each criterion needs a test. Every test is named after its criterion. verify:ac-trace fails the merge request if a criterion has no passing test.
  3. The tests are mutation-checked. critic:mutation changes the lines the merge request touched. A surviving mutant fails the merge request as a failing test. A critic agent must kill it without weakening a test, and guardrails fails any test that is deleted, skipped or loosened.

Of the three layers, only the mutation gate has failed on a public run so far. The failure paths of guardrails and verify:ac-trace are covered by unit tests only.

GitLab's Hands-off example, step by step, with our hero run (issue #3, GET /payments/{id} with a refund summary, 2026-10-19):

  • "You create an issue." Done once. The template's /assign line starts our Duo flow within a second. The owner's coding assistant filed #3 from his account, at his request. The owner filed the game-day issues (#4, #6) himself, on camera.
  • "Agents review it." Done for the issue: the planner reads it, checks that it is in scope and writes 8 numbered acceptance criteria before any code. Not done for the code: no AI reviews the merge request. The flow's service account cannot start Duo Code Review (lab MR !12), and our role cannot turn on automatic review.
  • "Create 1+ MRs." Done. The builder writes the code and the tests; a script opens MR !2 at T+3m57s.
  • "Run security scans." Done on every push: GitLab SAST, Secret Detection and Container Scanning, plus our exposure probe and secure:gate.
  • "Fix what they find." Done, in three rounds. Round 1: the probe found the open test routes (4 findings), and 1 mutant survived on the builder's if (!isRefund) guard. The critic closed the routes at the source, added tests and deleted that redundant guard. Round 2: 1 mutant survived that no test can kill; a fresh critic removed the redundant page ? null : check. Round 3: every gate passed, 29 of 29 mutants killed, 18 tests named after the 8 criteria. GitLab's scanner jobs ran green in every round, and none of their findings blocked the merge: the final secure:gate lists 433 findings, 0 of them blocking under our policy. The blocking findings came from our probe and the mutation gate.
  • "Merge when everything passes." Done, another way: GitLab refuses auto-merge to the flow, so a script pushed the verified head to main at T+16m29s (see Challenges).
  • "Deploy to staging." Done at T+18m37s, on Cloud Run, after a cosign signature check.
  • "Validate." Done. validate:staging runs the change's own live acceptance tests and a 26-test sample. Then a 0% canary is smoke-tested.
  • "Promote to production." Done at T+23m48s. The two-minute watch passed (error rate 0, p95 226 ms).
  • "Post a summary on the original issue." Done at T+28m24s: a receipt written by a script, not an LLM, with the nine stages (links checked signed out) and 28 events, each with its rule.
  • "You touched nothing." Human actions: 1. Approvals: 0. Manual jobs: 0. The owner's next push to main (docs) came 2m18s after the receipt; one of its commits, 80dc52a, was written during the run.

The nine stages, one public link each (hero run):

  • Plan: issue template, Assign trigger, 8 criteria, /estimate and /spend.
  • Create: Duo custom flow, agent commits, MR opened by the flow.
  • Verify: JUnit report and coverage, mutation critic (29 of 29 killed), HTTP tests on the image, AC trace.
  • Package: container registry, provenance gate, cosign, Artifact Registry, CycloneDX SBOM.
  • Secure: SAST, Secret Detection, Container Scanning, exposure probe, security gate.
  • Release: environments, 0% canary, promotion, GitLab Release.
  • Configure: Cloud Run specs in the repo, applied by CI over WIF; drift check.
  • Monitor: SLO watch; rollback and incident on game day (below).
  • Govern: guardrails, policy as code, signature checks, stage ledger, receipt.

Our game day (issue #6, filed by the owner himself, on camera): the flow merged in round 1, and the canary ran with 1.5 s of delay added on purpose. The watch failed (p95 1569 ms against 800 ms). CI put all traffic back on the hero release by T+18m40s, and the flow opened incident #7. Receipt at T+20m29s: "rolled back automatically". Human actions: 1. Approvals by people: 0. Manual jobs: 0. Stages: 9/9. The pipeline is red on purpose.

How we built it

  • The flow (flows/pocketful-autopilot.yml) is a GitLab Duo custom flow with 18 components: 4 agent components for 3 LLM seats (planner, builder, critic, plus a fresh critic for round two) and 14 script steps. It has no human-input step. Scripts push, open the merge request, wait, collect failures, merge and write the receipt. No LLM waits, merges or counts.
  • GitLab Duo features used: the custom flow, its service account, the Assign trigger, built-in tools (get_issue, file tools, run_tests, get_job_logs, gitlab_api_get, create_issue, create_issue_note) and a project agent config with our own flow image.
  • The pipeline has 11 stages. It uses GitLab's SAST, Secret Detection and Container Scanning templates, plus our own gates in tools/ and ci/: mutation (JUnit output), exposure probe, security gate (policy as code), AC trace and guardrails (agents may change only stage-4/).
  • Supply chain. package:promote signs only the digest that gate:passed recorded (keyless cosign) and copies it to Artifact Registry. Every deploy verifies the signature first. A CycloneDX SBOM is kept.
  • Google Cloud. Cloud Run staging and production, specs in deploy/cloudrun/, applied by CI through Workload Identity Federation. No service-account key exists.
  • Tests of the automation. Scripts are tested offline against a mock GitLab API, and full rehearsals ran in a lab project. At commit 69a42fb (2026-10-21): 310 app tests, 112 CI-gate tests, 72 deploy-helper tests, 267 flow-script tests and 43 mutation-tool tests pass, plus 46 pipeline-structure checks and the flow linter (18 components). The submitted commit, 9c61cdf (tag submitted-2026-10-23), and the commits after it change only documentation, so the code is the same.
  • AI tools. Built with Claude Code as a coding assistant (it also filed issues #1 to #3 in pocketful from my account, at my request); the in-product agents run on GitLab Duo.

Challenges we ran into

  • Exit codes do not route. A flow step reports "success" even when its command fails. So each script prints one word and the router matches that word.
  • The flow cannot use GitLab auto-merge (REST 403, GraphQL "no permission"). It pushes the verified head to the protected main instead. That is detected, not prevented: an LLM seat could push to main too.
  • Sessions end at about 60 minutes, and the platform dropped 3 of our 17 runs (lab #9, lab #10 and pocketful #2). After a drop no flow step runs, so there is no receipt. We filed a new issue and ran again.
  • A mutant no test can kill stopped issue #1 at the round cap. The critic guide now says: if no test can kill a survivor, simplify the line. The hero run did exactly that.
  • Our own gate refused a good deploy (pocketful #4, game day 1). GitLab made two merge request pipelines for one push, and the image tag pointed at the other pipeline's digest, so package:promote refused. It failed closed: production was untouched. The receipt also counted the duplicate pipeline as a person (Human actions: 2). Both are fixed, and game day 2 (#6) ran on the fix.
  • Event triggers stopped firing on 2026-10-14. The Assign trigger works, so the issue template assigns the issue to the flow in the same request.

Accomplishments that we're proud of

  • The gate catches agent mistakes, not only old ones. Measured on 2026-10-19 over five flow runs (lab #17, #18, #19 and pocketful #1, #3), the mutation gate stopped 5 surviving mutants: 1 left by the builder's own tests and 4 on lines a critic had changed. The critics removed 4 of them. The run that still had the fifth (#1) merged nothing.
  • An honest record. In the submission project, #1 stopped safely at the round cap; #2 was a platform drop with no receipt; #3 shipped in round 3 of 3 (9/9); #4 merged, then our provenance gate refused the deploy (fail closed, our bug, fixed); #6 was rolled back by design (9/9). All 17 runs (12 in the lab, 5 here): 5 shipped, 2 rolled back on purpose, 1 deploy refused, 4 stopped with "needs a human" before any merge, 3 platform drops, 2 failed to start. Merges with a red gate: 0. We have no time or cost baseline yet: we did not time a person doing the same change.
  • Agents cannot weaken their judges through a merge request. Every gate, policy and guide is outside the folder agents may change. (A direct push to main is the open gap: see below.)

What engineers said

We asked three engineers to watch the draft video, read the README and open the hero receipt. They replied in their own words and agreed to be quoted. (In Hasala's quote, "[one]" replaces "zero" with his OK: the receipt counts one human action, filing the issue.)

"The automated receipt tracking [one] human action alongside strict mutation testing is super impressive and addresses the biggest bottleneck in AI-assisted coding: verifying that AI-written tests actually test something meaningful. I would definitely trust this workflow for well-defined, non-breaking changes, Especially with the safety net of automatic rollbacks. For broader adoption on complex production systems, my biggest worry would be silent logical regressions that pass unit tests but break cross-service contracts, so integrating automated E2E staging checks before the final rollout would be my next wish-list item." — Hasala Jayasuriya, Magento2 Certified Developer | Adobe Certified Expert | Senior Tech Specialist

"Using deterministic mutation testing to validate AI-generated tests is a brilliant approach—it solves the classic 'who watches the watchmen' problem when an agent writes both the implementation and the test suite. While I'd readily trust this pipeline for isolated bug fixes, minor UI tweaks, or low-risk microservices, I would still hesitate to give it full autonomy over core business logic or database schema migrations without a human approval gate. To use this in a real production environment, I'd love to see post-deployment canary analysis or automated observability metrics integrated directly into that rollback trigger." — Kasun Karunathilake, AI Solutions | Databricks | GenAI & ML | Cloud Data Transformation | Technical Leadership & Delivery

"The standout feature here is definitely combining mutation testing with deterministic security gates. Usually, AI agents write superficial tests just to pass the pipeline, so forcing the agent's tests to survive mutation testing before shipping to Google Cloud is a game-changer. My main concern in a real-world setup would be how the agent handles ambiguous issue descriptions—if the initial prompt in the GitLab issue has a logical gap, the AI might build and test the wrong thing flawlessly. Adding a pre-code 'spec verification' step would make this production-ready for my workflows." — Isuru Devinda, Frontend Developer at Johnson & Johnson MedTech specialising in React

Where we stand on their concerns:

  • Scope. Hasala and Kasun said they would trust it for well-defined, low-risk changes. Kasun would still want a human approval gate for core business logic and schema migrations. That matches the design: the template asks for one additive change.
  • Staging and rollout checks. Staging validation, a 0% canary and a 2-minute SLO watch (errors, p95, version) with automatic rollback exist today (game day #6). Cross-service contract tests and Cloud Monitoring metrics as rollback inputs are not there yet.
  • Ambiguous issues. The planner writes numbered acceptance criteria and posts them on the issue before any code, and the criteria are frozen after the first push. Nobody reviews them before the build, though: that is the "spec verification" gap.

What we learned

  • Keep the LLM where judgement is needed: planning, writing code, fixing a reported failure. Waiting, merging, counting and reporting are scripts.
  • A critic needs something concrete. A surviving mutant with a file and a line is a good target. "Review this code" is not.
  • Measure the platform first. Most design changes came from small probe flows with no LLM.
  • Proof matters as much as automation. A receipt with signed-out links makes "hands-off" checkable.

What's next

  • Close the push-to-main gap. Keep the CI configuration in a separate protected project, or use a pipeline execution policy, and bind the Google Cloud trust to it.
  • Reach. Publish the gates as GitLab CI/CD Catalog components, make the probe routes config-driven, and plug in other mutation tools (Stryker, PIT, mutmut).
  • Incidents after the session. CI rolls back after the flow's session ends, but only the flow opens the incident. A project token would let CI open it.
  • Faster runs. The hero run took 28m24s from issue to receipt.
  • GitLab Duo Code Review on every flow merge request, once a flow can start it. A review a person asks for already works (lab: Duo reviewed and approved a flow MR).
  • What the engineers asked for (above): an optional human approval gate for risky changes, a spec check of the planner's criteria before the build, cross-service contract tests on staging, and Cloud Monitoring metrics as rollback inputs.

Built With

  • cosign
  • cyclonedx
  • docker
  • gitlab
  • gitlab-ci
  • gitlab-container-registry
  • gitlab-container-scanning
  • gitlab-duo-agent-platform
  • gitlab-environments
  • gitlab-releases
  • gitlab-sast
  • gitlab-secret-detection
  • google-artifact-registry
  • google-cloud-monitoring
  • google-cloud-run
  • node.js
  • pytest
  • python
  • ruby
  • semgrep
  • sigstore
  • typescript
  • workload-identity-federation
Share this project:

Updates

Submission history