Itemator

An AI agent fixed one bug and also added a 1,888-line install script nobody asked for. Itemator makes GitLab Duo hold the merge request until a person answers for every file like that.

Comment

Inspiration

In July 2025, GitHub Copilot opened a pull request in the Model Context Protocol's C# SDK to fix one deserialization bug. It also committed dotnet-install.sh: a 1,888-line install script the issue never mentioned. Two hours and 22 minutes after that commit, the maintainer reviewing it wrote:

"@copilot this file looks unnecessary, please delete it." (the comment)

That time, the reviewer spotted it. Itemator makes sure someone is asked every time, before the merge.

I build alone, and an AI coding agent writes most of my merge requests. I learned one thing the hard way: when an agent tells me it has done something, it isn't true until I have checked it. What I kept missing was never the code I asked for. It was what came along with it: an extra file, a new package, one changed line in the CI config.

Then October made it worse...

Patch week: October 2026

12 Oct  GitLab 19.4.2   critical patch release, self-managed   CVE-2026-97229 (reached through a CI/CD configuration)
21 Oct  GitLab 19.5.1   critical patch release, self-managed   CVE-2026-110264
21 Oct  PHP 8.3.36 and sibling releases                        CVE-2026-109618, CVE-2026-109619
        php.net: unauthenticated remote code execution; "an exploit had been demonstrated"

GitLab.com was already running GitLab's fixes; the two events are unrelated. But every team that runs PHP or self-manages GitLab had to ship a patch fast, and a patch written in a hurry, often by an agent, and reviewed in a hurry, is exactly where an unasked file rides along.

Itemator does not decide whether a version is safe. It makes sure a person answers for everything around the fix. Two staged runs, made by hand on my accounts for the demo, with the card's real words:

1. A security fix that also downloads and runs code (staged !17). The issue names both PHP CVEs. The change bumps the Dockerfile to PHP 8.3.36, and also adds docker/install-ext.sh. The card:

`docker/install-ext.sh` (outside the issue's scope): changes docker/install-ext.sh, more than the
version change; the issue names CVE-2026-109618, CVE-2026-109619, whose fix is the version change
in Dockerfile. [...] To drop it from this MR, reply `remove`.

What it does, as the card's model read the diff (check it): A new shell script of three lines:
a shebang, a comment about PHP image extensions, and a command that downloads a script from a
URL and pipes it to sh.

I answered remove; Duo took the script out, and only the fix was left in the merge request.

2. Production patched, staging not (staged !20). The Dockerfile moves to the fixed PHP. A staging file doesn't:

 # Dockerfile (the fix)
-FROM php:8.3.35-cli
+FROM php:8.3.36-cli
 # deploy/compose.staging.yml (not in the change: still on the old PHP)
     image: php:8.3.35-fpm

The card names that file, and offers one word to fix it:

`deploy/compose.staging.yml` (still on the old version): still pins php 8.3.35 (image, line 4),
while this change moves php to 8.3.36 in Dockerfile. [...] To move it to the new version too,
reply `bump`: in `deploy/compose.staging.yml`, `image: php:8.3.35-fpm` becomes
`image: php:8.3.36-fpm`.

bump, and Duo moved it in one commit (on a second try, after I fixed a fault the first run found). This is not hypothetical. Of 24 public agent pull requests that moved a version, 9 left another file on the old one (5 read as clearly real), and no reviewer named any. And in this very week's real PHP fixes, 2 of 32 that moved a PHP pin left a copy behind (both opened from people's accounts); one, merged as "critical PHP RCE fixes", still builds a second, command-line image from the unfixed PHP today.

The rest of the patch-week guards:

  • A stopgap that ends with its fix. Answer until #N; the merge request that closes #N asks whether to take the workaround out (staged !21 → !22).
  • Only the team answers. On a public project anyone can comment, but only a project member's answer counts. A CI file the team quieted is asked about again if the change pipes a download into a shell (staged !25).
  • A self-managed server still on an unpatched version. The job asks GitLab's CI lint for the pipeline a change produces; on a version these patch releases fix, it does not send the change's CI file there, and says why (code and tests, not a run on an unpatched server).
  • Known malware by name. When OSV has a malware report for a new package, its thread says so first, even after the registry took it down: 200 of 200 real reported PyPI names.

What it does

The !7 thread: the card asks about .gitlab-ci.yml, the author replies always, and the flow opens policy merge request !8

Itemator holds a coding agent's merge request as a draft until a person answers for each file its issue never asked for. We call those files extras.

  1. A CI job finds them with fixed rules, no model: a new dependency, a CI or config file, a deleted or weakened test, a binary, a git hook, a file that instructs coding agents.
  2. A GitLab Duo flow asks: one thread per extra, on the line it changed, and the merge request goes to Draft.
  3. You answer with one word, and Duo does exactly that: the file's name keeps it, remove or split moves it out, bump moves an old copy to the new version, until #N keeps a stopgap until its fix lands.
  4. Answer always once, and Duo opens a one-line policy merge request. Once a person merges it, the next agent merge request is not asked about that file.

That last step is live, not staged: GitLab's own Developer Flow opened !7; I typed always; policy !8 was merged on my account by my coding assistant through the API, as the test planned; the next agent merge request, !9, asked nothing.

The card on !9: the team policy quieted one extra, .gitlab-ci.yml, and no thread

Path A, Assisted. The rules describe Assisted as flows that "propose actions and wait for your go-ahead before proceeding". Here the go-ahead is one typed word per file, and every merge is left to a person.

How it spends GitLab Duo

The expensive part, reading the whole diff and deciding what is beyond the issue, is done by fixed rules with zero model calls. Duo does the acting:

  • Deciding what is an extra (packages, advisories, malware, CVE scope, old copies): the CI job, 0 model calls, 0 GitLab Credits.
  • The card and its threads: about 6 model calls (9 tool calls) with three extras, at most 1.9 GitLab Credits at list price; my meter charged 0.4 to 0.8 of that. I cut it from 9 model calls with a registered prompt edit.
  • Carrying out the answers: 3 to 34 tool calls, at 0.13 to 0.25 GitLab Credits per tool call on my meter.
  • A question an always answer already settled: nothing at all. On public agent pull requests, up to half of all questions (297,925 of 597,934, an upper bound) repeat one the same repository was already asked.
  • For scale, GitLab's Developer Flow writing a change took 20 to 27 tool calls. The card that checks it took 9.

The hackathon's monthly allowance covers a lot of reviewing:

$$\frac{500 \text{ GitLab Credits}}{6 \text{ model calls} \times 0.3125 \text{ credits per call}} \approx 266 \text{ typical cards}$$

One thing that did not help, and I report it: handing Duo precomputed facts did not cut its work (16 model calls with them, 14 without). These are estimates from one account's meter, which lags by hours; not a bill.

Check it yourself, no login

  1. Paste a diff and its issue: the page lists what the CI job would. Nothing you paste leaves it.
  2. The #645 example, replayed (staged): the card names dotnet-install.sh, I answer remove, Duo takes it out. The first flow session failed and I typed the answer again.
  3. The always loop (real): one question on !7, policy !8, nothing asked on !9.
  4. What shipped: every deployment, its extras, and who answered each.

For the judges

  • Uses Duo Agent Platform (video 0:17): two custom flows from the AI Catalog with their own service accounts and triggers, two Agent Skills; GitLab's Developer Flow wrote the agent merge requests (GitLab lists the person who started it as author). Where: flows/, skills/.
  • Technological implementation (video 0:23): the check is injected by a pipeline execution policy, so a merge request cannot edit or skip it: 0 of 11 bypass attempts merged; 784 of 784 one-line mutants caught.
  • Design (video 0:49, 1:34): issue → agent merge request → injected job → card → one-word answer → Draft lifted → a person merges → deployment, package, What-shipped page, a note on the issue.
  • Potential impact: We sealed this test before the outcome and it went against us; we report it first because the list's job is to ask, and the ask is cheap (one reply). The forward test found listed files acted on no more often (20 of 245, 8%) than unlisted ones (227 of 1,593, 14%). What holds: its frozen rules list 24 files real reviewers objected to in public agent pull requests, dotnet-install.sh among them; reviewers took a median of 3 hours to comment, and 8 of the 24 merged anyway. It asks on about one agent pull request in seven (10,712 of 71,582), and a wrong question costs one reply, not an extra.
  • Innovation (video 1:52): Code Owners routes every change to a path's owner, asked for or not; review bots score a change against its ticket. Itemator asks only about what the issue did not ask for, carries out the answer, and remembers it as reviewed policy.
  • Most Stages Covered: all nine stages touched, each linked in the README's stage table; Monitor is the thinnest and says so.

How I built it

A pipeline execution policy injects the check into every merge request pipeline. Flow triggers start the two Duo flows. Every merge to main records a deployment, uploads the checker to the package registry and rebuilds the What-shipped page. Every number in the README links to the results file it comes from, and a script fails the build if one does not match.

Challenges

  • Outside reviewers fooled it. A curl piped to sh inside source code, a test changed to assert True, a pip install inside a Dockerfile: each slipped past at first. Each is now caught, and every trick is now a test.
  • Two things only showed up live. A flow's token cannot read GitLab's members API, and a CI job token cannot read issue comments on gitlab.com. Unit tests with a fake API had passed. Both are fixed.
  • False alarms are real. On merge requests GitLab's Developer Flow merged into gitlab-org/gitlab in September 2026, the rules spoke on 9 of 84, and model labellers found a clear extra in none of those 9. Each cost one reply.

Accomplishments

  • The live always loop with GitLab's own Developer Flow: one question on !7, nothing asked on !9.
  • A person from outside the project answered on !278 in their own words, and the flow recorded it as theirs.
  • On 31 published security advisories about CI workflows, set aside before I wrote the rule, one check named 10; its second version, written after I saw that, named 15 and stayed quiet on 29 of the 31 fixes.
  • A forward test sealed before the outcome existed, published although it went against me.

What I learned

When reviewers object on GitLab's own merge requests, they object to code and tests, which the rules do not list. Itemator names the kinds of change a one-click approval skips; reading the code stays the reviewer's job. Not built yet: private projects (GitLab's job token cannot read issues there). The flow triggers are off outside my own runs and nothing starts by itself after the deadline, so a merge request a judge opens gets no card; the links above show it working.

Sustainability (entered for Most Environmentally Impactful)

No model reads the diff; the check's image went from 412 MB to 21 MB (median job 18.3 s); the card went from 9 model calls to 6; and an always answer removes a question and its model calls from every later merge request. Each new dependency's thread says what it downloads on every fresh install, so a reviewer can say no before the bytes ship: by the Green Software Foundation's SCI method, one remove of the live run's dependency avoids 0.014 g to 2.1 g CO2e per fresh install, a bracket because network energy per GB is disputed.

Built With

  • agent-skills
  • ai-catalog
  • crates.io
  • custom-flows
  • deps.dev
  • flow-triggers
  • gitlab
  • gitlab-ci
  • gitlab-duo
  • gitlab-duo-agent-platform
  • gitlab-package-registry
  • gitlab-pages
  • graphql
  • npm
  • nuget
  • osv
  • pipeline-execution-policies
  • pypi
  • python
Share this project:

Updates

Submission history