Slop Shield

A GitLab Duo flow that checks the code claims in a bug report at the version it names. Of 12 reports it closed 3 with evidence, proposed closing 5 and left 4 to read; no real bug was closed.

Comment

Inspiration

GNOME, whose code lives on GitLab, no longer runs its bug bounty for GLib, glib-networking and libsoup: of 298 reports, 71 were accepted, and the maintainer who asked to end it wrote, "I was overwhelmed with incoming AI-generated issue reports" (blogs.gnome.org, 2 Oct 2026). In the same post the maintainer argues against banning AI-generated reports, because "the overwhelming majority of vulnerability reports are AI-generated", so someone still has to sort them.

GitLab's own bug bounty policy says: "Due to an increase in low-quality AI-generated submissions, all reports must include one of the following reproduction artifacts [...] Reports without sufficient and verifiable evidence will be closed as N/A without further investigation [...]" Their Security Standard v1.0 puts it shorter: "Code is abundant. Trust is scarce."

Slop Shield is for maintainers and security teams who triage incoming reports. Whoever gets a report still has to open the code at the version the report names and check whether the function, the line and the value it describes are really there. The check is mechanical and slow, and nothing else can happen until it is done. We wanted GitLab to do it inside the project, as soon as the report is filed. For our demo library the judge reads the code itself at the claimed tag. For third-party code (curl, QEMU, GLib) it relies on facts from CI, so each report first needs a descriptor we write (its claims, which of them are key, and any versions to compare with), the code base pinned by commit in the project, and a run of the main pipeline.

What it does

A report is filed as an issue with the label report::incoming. Creating it starts a custom GitLab Duo Agent Platform flow with three steps:

  • gate (agent, read-only) takes the issue number from the trigger, never from the text, and lets a new report through only if it carries the incoming label;
  • judge (agent) weighs every claim (function, file, line, value, call path) against facts collected at the exact version the report names, posts one note with a table (claim, key or not, where it was checked, result, evidence) and a Read: list of everything it read, and sets labels;
  • closer (deterministic step, no model) closes the issue, and the only way into it is the judge's exact answer CLOSE.

There are five outcomes. Only one closes on its own: invented, when a key claim is refuted with evidence. Wrong version and conclusion does not follow get a proposed close, and a person decides. Confirmed gets needs-security, and severity stays with a person. Anything unsettled goes to needs a human. A search that could not have found something is not allowed to count as proof that it is missing.

For a confirmed report on our demo library (bug planted on purpose), the flow opens a confidential tracking issue. On a rehearsal of the same report, CI reproduced the bug in a container with no network, a fix was merged outside the flow (pipeline red with the test, green with the fix), and CI published a patch release with a Security section.

A public board on Google Cloud Run shows the counts, the issues, the red and green pipelines and the release, read from the public GitLab API only.

In the clean run on 2026-10-20, 12 reports (9 real public reports about curl, QEMU and GLib, retold as claims, and 3 synthetic ones about our own library) went from 12 to read in full by hand to 4 in 3 min 50 s. Three invented reports were closed with evidence, five were proposed for closing for a person to confirm, and no real bug was closed. The maintainers decided 8 of the 9 real reports; Slop Shield, tuned on these reports, went the same direction on 7 of those 8. The eighth, a real curl CVE, was handed to a person because one hop of its call path goes through a function pointer, and our checker does not follow pointers yet.

Six of the nine real reports come from a public list of "AI slop security reports submitted to curl". curl's maintainers closed all six as not applicable or spam, and Slop Shield found none of them valid: it closed two with evidence and proposed closing the other four.

How we built it

  • The Duo flow (flows/slop-shield.yml, AI Catalog version 1.3.0): two agent steps and one deterministic step, started by the Work item: created trigger, with a Mention trigger set up for re-checks (used in a rehearsal). Write tools are pinned to this project, the judge's update_issue can change labels only, create_issue always creates a confidential issue, and no tool in the flow can run code, push commits or open merge requests.
  • Facts come from CI, without a model. curl lives on GitHub and GLib on gitlab.gnome.org, and the flow's tools address projects on GitLab.com only (we did not try other hosts). A CI job fetches each code base at a full commit SHA pinned in targets.yml, and claim-check checks the claims listed in each report's descriptor in data/reports/ inside unshare -rn with no network, after proving the network is blocked. Every report gets a positive control (a known symbol must be found) and a negative one (a made-up string that must not be found). The judge also reads files and trees itself on GitLab.com: our demo library at the claimed tag, and QEMU at the commit pinned in the CI facts.
  • Test, fix and release ran in a rehearsal of the synthetic report SS-S01, before the clean run. The repro job, set up for SS-S01, runs a minimal test written from the report's claims, never the reporter's code, in docker run --network none --read-only --cap-drop ALL, after five controls. MR !6 went red with the test and green with the fix; after the merge the release job published v0.1.1 with CWE, affected and fixed versions, and a suggested score for a maintainer to decide.
  • The regression test and the fix in MR !6 were written by an AI coding assistant session (Claude Code), which also merged the MR from the author's account, outside the flow. The flow has no tool that writes code, pushes or merges.
  • Why Supervised: the flow closes a report on its own only when a key claim is refuted with evidence, and that close says "Reopen if this is wrong." On every other outcome it only posts its note and sets labels (and, for a confirmed report on our own library, opens a confidential tracking issue); a maintainer decides a proposed close, a report that needs a human, and the severity of a confirmed report.
  • The board uses the Python standard library only and no token, and is deployed to Cloud Run (scale to zero, one instance at most). Deployment code is in board/Dockerfile and deploy/cloudrun.sh.
  • The code, tests and documentation were written with AI coding assistants (Claude Code), directed by the author.

Challenges we ran into

  • gitlab_blob_search called without ref searched only the default branch, so text that exists only on a tag came back as 0 results with no error, which reads like "this code does not exist". We read files and trees with ref instead.
  • Tool options can fix a parameter's value but cannot restrict a path, so "never read the expected answers in data/" is a prompt rule. That is why every note lists what it read, and we checked all 12 sessions after the run: none read them.
  • A marker string from a confidential issue the flow read showed up in the log of the flow's workload job. Anonymous users get 401/403 there; project members can read it. So the confidential issues in this demo hold synthetic material only. Not reproduced with a separate minimal test.
  • The "Developer + AI" role could create and enable a custom flow and add triggers, while GitLab's docs for custom flows say enabling needs Maintainer or Owner. Not reproduced with a separate minimal test.
  • A real curl CVE has a call path with a function-pointer hop. Before the clean run we changed our expected outcome for it from confirmed to needs a human, rather than relabel which claims count as key.
  • The 12 reports are the same ones we built and tuned the flow on in rehearsals, so the numbers are a regression check on known reports, not a blind test. A larger, unseen set is the next step.
  • One report we expected to close (SS-R02) was only proposed: the named function does not exist, but a similar name (Curl_ldap_err2string_a) made the checker call it a mismatch. The report stayed open, which is the safe direction, and the README lists it as a known miss.

Accomplishments that we're proud of

  • 0 real bugs closed by mistake; every closure carries a refuted key claim and its evidence in the issue.
  • 12 to 4 reports to read in full by hand, plus 5 proposed closes a person confirms from the note; each report took about 72 s from trigger to verdict, with sessions running in parallel.
  • Every result is a GitLab object a judge can open: issues, labels, notes, pipelines, jobs, a merge request and a release (the agent sessions and the confidential tracking issue are visible to project members only).

What we learned

Reading code at the right version settles a lot of slop on its own: functions that do not exist, lines past the end of the file, a real line at a different version. What it cannot settle is behaviour, so "confirmed" in Slop Shield means the code is there, not that it is exploitable.

What's next

  • Tell a function that does not exist apart from one with a similar name (the SS-R02 miss).
  • Follow a function pointer when it is assigned in the same file (the SS-R07 hop).
  • Run it on a larger set of public reports.

Built With

  • docker
  • gitlab
  • gitlab-ci
  • gitlab-duo-agent-platform
  • google-cloud-run
  • python
Share this project:

Updates

Submission history