Pocketful Autopilot is for small teams that ship a payments API with no QA, security or SRE person. AI writes their code and their tests, so who tests the tests? Our imported app had 99.70% line coverage, yet mutation testing found 13 changes that no test noticed. A test route returned every password hash. We left it open on purpose, as a job for the agents. With Autopilot, one GitLab issue starts a Duo flow, and agents fix what the CI gates find, including a mutation gate. In our hero run the change reached production with 1 human action and 9 of 9 stages, and the receipt came at 28m24s. After we submitted, three runs with 1 human action each: two shipped, and #9 was rolled back automatically on a false positive (one probe timed out). #11 then shipped its own change and #9's.

Path B · Hands-off · Cloud Run + Workload Identity Federation · 9 of 9 stages

Path B: what we started from and what we added. We started from Pocketful, a small payments API (TypeScript on Node 24, no npm dependencies) in our MIT-licensed repo four-seat-dark-factory, commit efb9f4f of 2026-10-05, imported on 2026-10-13 (NOTICE lists what came over). Everything else was built in the Submission Period: the GitLab Duo custom flow (4 AI agents, 14 script steps), the CI gates, the Cloud Run deploy over WIF with canary, SLO watch and rollback, the receipt and the docs. People changed the app only to run it on Cloud Run and to add test helpers (2026-10-13 to 10-15). Every later change to the app is an agent commit from a run: #3,

4, #6, #8, #9 and #11.

Verify it in 5 minutes (no sign-in needed):

  • Hero receipt on issue #3: "Human actions: 1", Approvals 0, Manual jobs 0, 9 of 9 stages with links.
  • MR !2: three agent commits, one per round, merged by the flow's service account.
  • Hero main pipeline 2962311393: all green, no manual job (monitor:rollback skipped: the watch passed).
  • Game-day main pipeline 2971404006: red on purpose. monitor:watch failed (p95 1569 ms against 800 ms), then monitor:rollback passed.
  • Incident #7: opened by the flow. The owner closed it about five hours later, which lets runs start again.
  • Live /health: with curl -si, the x-app-version header is e154b5b0..., run #11's commit. The hero release afecd876 served production until run #8 (2026-10-24), apart from game day #6.
  • README: the run diagram, the gates and "Honest limits".

Before you check, five things that can look wrong:

  • Names. MR !2 shows the owner as its author, and the flow's commits show his name. This is composite identity: a Duo flow acts for the person who started it. The commits' email and the merge belong to the service account. Until the receipt, issue #3 shows the owner only at 08:38:48, the second it was filed.
  • Environments page. Production's last successful deployment is release:promote on e154b5b0 (run #11). The rollback and incident in the hero receipt's Monitor row ran in game day #6 and run #9, whose main pipelines are red.
  • Staging keeps its /_test routes on, on purpose, with throwaway data. Production has them off (404).
  • "Skipped" pipelines are the owner's pushes with ci.skip: to main (docs, setup and fixes between runs) and the tag submitted-2026-10-23. The Duo session page needs a sign-in.
  • Two times for one deployment. The receipt's event list shows when a deploy job started, its Timeline when it ended. This page uses the Timeline.

Inspiration

Pocketful is a small payments API. An AI agent team built it in an earlier hackathon: 248 tests, all green. Then we asked who checks the checks. Mutation testing changed the code in 180 small ways in two files, and 13 of those changes survived. A test-only route, GET /_test/export, needed no login and returned every user's email and password hash.

That route breaks a house rule that we wrote for this project: test-only routes are off unless POCKETFUL_TEST_HOOKS=1 (AGENTS.md section 5, in the repository since 2026-10-13). The imported app does not follow it. We left it that way on purpose, as a job for the agents. The team writes the rule once. The gates enforce it. The agent applies it.

A green pipeline only proves that tests ran. We wanted the tests checked inside GitLab, with no human in the middle, and with a receipt that proves what happened.

Mutation testing is not new: Stryker and PIT exist, and Google shows surviving mutants in code review. Mutation-guided LLM test generation exists too: Meta's ACH (arXiv 2501.12862) and the MuTAP research method (arXiv 2308.16557). What we add is where the mutant sits. A surviving mutant is a failing merge gate inside a hands-off GitLab Duo flow. It shows up as a failing test in the merge request, and the agents must beat it before anything ships. (Google: Petrović and Ivanković, ICSE-SEIP 2018; Petrović, Ivanković, Fraser and Just, arXiv 2102.11378.)

What it does

One person files one issue. GitLab does the rest, then proves it. Autopilot tests the tests in three layers:

  1. Criteria are frozen at the first push. A planner agent writes 3 to 8 numbered acceptance criteria. The guardrails job fails any later change to them.
  2. Each criterion needs a test. Every test is named after its criterion. verify:ac-trace fails the merge request if a criterion has no passing test.
  3. The tests are mutation-checked. critic:mutation changes the lines the merge request touched. A surviving mutant fails the merge request as a failing test. A critic agent must kill it without weakening a test, and guardrails fails any test that is deleted, skipped or loosened.

Of the three layers, only the mutation gate has failed on a public run so far. The failure paths of guardrails and verify:ac-trace are covered by unit tests only.

Impact: a coverage gate would have merged it. In hero round 2 (pipeline 2962276799) every gate passed except critic:mutation: 286 tests, 99.71% of lines covered. Its one survivor, like round 1's, was a redundant check, not a wrong answer. The gate checks only changed lines, so it has not touched the 13 old survivors.

GitLab's Hands-off example, step by step, with our hero run (issue #3, GET /payments/{id} with a refund summary, 2026-10-19):

  • "You create an issue." Done once. The template's /assign line starts our Duo flow within a second. The owner's coding assistant filed #3 from his account, at his request. The owner filed the game-day issues (#4, #6) himself, on camera.
  • "Agents review it." Done for the issue: the planner reads it, checks that it is in scope and writes 8 numbered acceptance criteria before any code. Not done for the code: no AI reviews the merge request. The flow's service account cannot start Duo Code Review (lab MR !12), and our role cannot turn on automatic review.
  • "Create 1+ MRs." Done. The builder writes the code and the tests; a script opens MR !2 at T+3m57s.
  • "Run security scans." Done on every push: GitLab SAST, Secret Detection and Container Scanning, plus our exposure probe and secure:gate.
  • "Fix what they find." Done, in three rounds. Round 1: the probe found the open test routes (4 findings), and 1 mutant survived on the builder's if (!isRefund) guard. The critic closed the routes at the source, added tests and deleted that redundant guard. Round 2: 1 mutant survived that no test can kill; a fresh critic removed the redundant page ? null : check. Round 3: every gate passed, 29 of 29 mutants killed, 18 tests named after the 8 criteria. GitLab's scanner jobs ran green in every round: the final secure:gate lists 433 findings, 0 of them blocking under our policy. The blocking findings came from our probe and the mutation gate.
  • "Merge when everything passes." Done, another way: GitLab refuses auto-merge to the flow, so a script pushed the verified head to main at T+16m29s (see Challenges).
  • "Deploy to staging." Done at T+18m37s, on Cloud Run, after a cosign signature check.
  • "Validate." Done. validate:staging runs the change's own live acceptance tests and a 26-test sample. Then a 0% canary is smoke-tested.
  • "Promote to production." Done at T+23m48s. The two-minute watch passed (error rate 0, p95 226 ms).
  • "Post a summary on the original issue." Done at T+28m24s: a receipt written by a script, not an LLM, with the nine stages (links checked signed out) and 28 events, each with its rule.
  • "You touched nothing." Human actions: 1. Approvals: 0. Manual jobs: 0. The owner's next push to main (docs) came 2m18s after the receipt; one of its commits, 80dc52a, was written during the run.

The nine stages, one public link each (each in detail):

  • Plan said no in lab #21: a change freeze, before any LLM call.
  • Verify failed hero round 1.
  • Secure blocked the same round.
  • Package refused #4's deploy and failed closed: our bug, fixed.
  • Monitor rolled back #6 (fault injected) and #9 (nothing injected); the flow opened incidents #7 and #10.
  • Create (no gate of its own; its round cap stopped #1), Release, Configure (drift check) and Govern (policy as code, stage ledger, receipt) have not been seen refusing on a public run.

Our game day (issue #6): the flow merged in round 1, and the canary ran with 1.5 s of delay added on purpose. The watch failed (p95 1569 ms against 800 ms). CI put all traffic back on the hero release by T+18m40s, and the flow opened incident #7. Receipt at T+20m29s: "rolled back automatically". Human actions: 1. Approvals by people: 0. Manual jobs: 0. Stages: 9/9.

Track record after submission (2026-10-24 and 25, UTC). The owner filed each issue from a link the owner's coding assistant had prefilled; #8 and #9 reuse texts that shipped as lab #18 and #23. All three passed every merge request gate in round 1. Each receipt: Human actions: 1, approvals 0, manual jobs 0, 9/9.

  • #8: shipped, 21m37s from issue to receipt.
  • #9: rolled back automatically, 16m51s. One /health probe timed out after 5 s, and Cloud Run's request log (not public) has no entry for it: a network fault by our reading, not the change. With at most 72 requests in the watch, one failure always breaks the 1% error budget. The flow opened incident #10; the owner closed it 25h54m later with a note on the cause: a human action outside the runs, which the incident asks for.
  • #11: shipped, 16m27s, and took #9's change to production.

How we built it

  • The flow (flows/pocketful-autopilot.yml) is a GitLab Duo custom flow with 18 components: 4 agent components for 3 LLM seats (planner, builder, critic, plus a fresh critic for round two) and 14 script steps. It has no human-input step. Scripts push, open the merge request, wait, collect failures, merge and write the receipt. No LLM waits, merges or counts.
  • GitLab Duo features used: the custom flow, its service account, the Assign trigger, built-in tools (get_issue, file tools, run_tests, get_job_logs, gitlab_api_get, create_issue, create_issue_note) and a project agent config with our own flow image.
  • The pipeline has 11 stages. It uses GitLab's SAST, Secret Detection and Container Scanning templates, plus our own gates in tools/ and ci/: mutation (JUnit output), exposure probe, security gate (policy as code), AC trace and guardrails (agents may change only stage-4/).
  • Supply chain. package:promote signs only the digest that gate:passed recorded (keyless cosign) and copies it to Artifact Registry. Every deploy verifies the signature first. A CycloneDX SBOM is kept.
  • Google Cloud. Cloud Run staging and production, specs in deploy/cloudrun/, applied by CI through Workload Identity Federation. No service-account key exists.
  • Tests of the automation. Scripts are tested offline against a mock GitLab API, and full rehearsals ran in a lab project. At commit 69a42fb (2026-10-21): 310 app tests, 112 CI-gate tests, 72 deploy-helper tests, 267 flow-script tests and 43 mutation-tool tests pass, plus 46 pipeline-structure checks and the flow linter (18 components). The submitted commit, 9c61cdf (tag submitted-2026-10-23), and the commits after it change no automation: they are docs commits and three agent commits (runs #8, #9, #11) that add endpoints and tests in stage-4/.
  • AI tools. Built with Claude Code as a coding assistant (it also filed issues #1 to #3 in pocketful from my account, at my request); the in-product agents run on GitLab Duo.

Challenges we ran into

  • Exit codes do not route. A flow step reports "success" even when its command fails. So each script prints one word and the router matches that word.
  • The flow cannot use GitLab auto-merge (REST 403, GraphQL "no permission"). It pushes the verified head to the protected main instead. That is detected, not prevented: an LLM seat could push to main too.
  • Sessions end at about 60 minutes, and the platform dropped 3 of our 20 runs (lab #9, lab #10 and pocketful #2). After a drop no flow step runs, so there is no receipt. We filed a new issue and ran again.
  • A mutant no test can kill stopped issue #1 at the round cap. The critic guide now says: if no test can kill a survivor, simplify the line. The hero run did exactly that.
  • Our own gate refused a good deploy (pocketful #4, game day 1). GitLab made two merge request pipelines for one push, and the image tag pointed at the other pipeline's digest, so package:promote refused. It failed closed: production was untouched. The receipt also counted the duplicate pipeline as a person (Human actions: 2). Both are fixed, and game day 2 (#6) ran on the fix.
  • A false alarm rolled back a change (#9, above). The watch still rolls back on one failed request. Next: retry a timed-out probe once.
  • Event triggers stopped firing on 2026-10-14. The Assign trigger works, so the issue template assigns the issue to the flow in the same request.

Accomplishments that we're proud of

  • The gate catches agent mistakes, not only old ones. Measured on 2026-10-19 over five flow runs (lab #17, #18, #19 and pocketful #1, #3), the mutation gate stopped 5 surviving mutants: 1 left by the builder's own tests and 4 on lines a critic had changed. The critics removed 4 of them. The run that still had the fifth (#1) merged nothing.
  • An honest record. In the submission project, #1 stopped safely at the round cap; #2 was a platform drop (no receipt); #3 shipped in round 3 of 3; #4 merged, then our provenance gate refused the deploy (fail closed, our bug, fixed); #6 was rolled back by design; #8 and #11 shipped; #9 hit a false alarm. All 20 runs (12 in the lab, 8 here): 7 shipped, 2 rolled back on purpose, 1 by a false alarm, 1 deploy refused, 4 stopped with "needs a human" before any merge, 3 platform drops, 2 failed to start. Merges with a red gate: 0 of 11. We have no time or cost baseline yet: we did not time a person doing the same change.
  • Agents cannot weaken their judges through a merge request. Every gate, policy and guide is outside the folder agents may change. (A direct push to main is the open gap: see below.)

What engineers said

We asked three engineers to watch the draft video, read the README and open the hero receipt. They replied in their own words and agreed to be quoted. (In Hasala's quote, "[one]" replaces "zero" with his OK: the receipt counts one human action, filing the issue.)

"The automated receipt tracking [one] human action alongside strict mutation testing is super impressive and addresses the biggest bottleneck in AI-assisted coding: verifying that AI-written tests actually test something meaningful. I would definitely trust this workflow for well-defined, non-breaking changes, Especially with the safety net of automatic rollbacks. For broader adoption on complex production systems, my biggest worry would be silent logical regressions that pass unit tests but break cross-service contracts, so integrating automated E2E staging checks before the final rollout would be my next wish-list item." — Hasala Jayasuriya, Magento2 Certified Developer | Adobe Certified Expert | Senior Tech Specialist

"Using deterministic mutation testing to validate AI-generated tests is a brilliant approach—it solves the classic 'who watches the watchmen' problem when an agent writes both the implementation and the test suite. While I'd readily trust this pipeline for isolated bug fixes, minor UI tweaks, or low-risk microservices, I would still hesitate to give it full autonomy over core business logic or database schema migrations without a human approval gate. To use this in a real production environment, I'd love to see post-deployment canary analysis or automated observability metrics integrated directly into that rollback trigger." — Kasun Karunathilake, AI Solutions | Databricks | GenAI & ML | Cloud Data Transformation | Technical Leadership & Delivery

"The standout feature here is definitely combining mutation testing with deterministic security gates. Usually, AI agents write superficial tests just to pass the pipeline, so forcing the agent's tests to survive mutation testing before shipping to Google Cloud is a game-changer. My main concern in a real-world setup would be how the agent handles ambiguous issue descriptions—if the initial prompt in the GitLab issue has a logical gap, the AI might build and test the wrong thing flawlessly. Adding a pre-code 'spec verification' step would make this production-ready for my workflows." — Isuru Devinda, Frontend Developer at Johnson & Johnson MedTech specialising in React

Where we stand on their concerns:

  • Scope. Hasala and Kasun said they would trust it for well-defined, low-risk changes. Kasun would still want a human approval gate for core business logic and schema migrations. That matches the design: the template asks for one additive change.
  • Staging and rollout checks. Staging validation, a 0% canary and a 2-minute SLO watch (errors, p95, version) with automatic rollback exist today (game day #6, run #9). Cross-service contract tests and Cloud Monitoring metrics as rollback inputs are not there yet.
  • Ambiguous issues. The planner writes numbered acceptance criteria and posts them on the issue before any code, and the criteria are frozen after the first push. Nobody reviews them before the build, though: that is the "spec verification" gap.

What we learned

  • Keep the LLM where judgement is needed: planning, writing code, fixing a reported failure. Waiting, merging, counting and reporting are scripts.
  • A critic needs something concrete. A surviving mutant with a file and a line is a good target. "Review this code" is not.
  • Measure the platform first. Most design changes came from small probe flows with no LLM.
  • Proof matters as much as automation. A receipt with signed-out links makes "hands-off" checkable.

What's next

  • Close the push-to-main gap. Keep the CI configuration in a separate protected project, or use a pipeline execution policy, and bind the Google Cloud trust to it.
  • Reach. Publish the gates as GitLab CI/CD Catalog components, make the probe routes config-driven, and plug in other mutation tools (Stryker, PIT, mutmut).
  • Incidents after the session. CI rolls back after the flow's session ends, but only the flow opens the incident. A project token would let CI open it.
  • Faster runs. Issue to receipt: 28m24s in the hero run (three rounds), 16m27s to 21m37s in the later runs.
  • GitLab Duo Code Review on every flow merge request, once a flow can start it. A review a person asks for already works (lab: Duo reviewed and approved a flow MR).
  • What the engineers asked for (above): an optional human approval gate, a spec check of the planner's criteria, contract tests on staging, and Cloud Monitoring metrics as rollback inputs.

Built With

  • cosign
  • cyclonedx
  • docker
  • gitlab
  • gitlab-ci
  • gitlab-container-registry
  • gitlab-container-scanning
  • gitlab-duo-agent-platform
  • gitlab-environments
  • gitlab-releases
  • gitlab-sast
  • gitlab-secret-detection
  • google-artifact-registry
  • google-cloud-monitoring
  • google-cloud-run
  • node.js
  • pytest
  • python
  • ruby
  • semgrep
  • sigstore
  • typescript
  • workload-identity-federation
Share this project:

Updates

Submission history