-
-
MR pipeline stages: verify, gate, package, secure, approve
-
MR !13, written by the implement-issue flow, merged by agent-dispatch at 17:51 UTC; issue #3 closed
-
MR !15 (staged): the gate finds a new skip in tests/test_cli.py and blocks under the agent profile
-
Scheduled agent-dispatch log: posts a note on blocked MR !15 instead of starting a fix
-
MR !6: the gate says review, and a person runs the manual human-review job
-
README: on the blind held-out v2 run the gate caught 20 of 26 cheats and blocked 0 of 20 honest fixes
-
MR !21: fix-pipeline pushes a one-line code fix with no test changed; agent-dispatch merges the green result
Inspiration
Teams now let agents fix failing pipelines and turn issues into merge requests. When a test fails, the cheapest way to a green pipeline is often to weaken the check: change the expected value, add a skip, wrap the assert in try/except, lower fail_under, add allow_failure: true, || true or # nosec. ImpossibleBench (2025) reports frontier agents modifying or special-casing tests that conflict with the spec. A reviewer can miss a one-line skip in a large diff. If agent merge requests merge without a person, something has to check the checks.
What it does
tamper-gate is a CI gate for merge requests. It diffs merge-base(base, head)..head, compares each Python test function before and after with ast (not line regexes), and reads pytest, coverage, warning-filter, GitLab CI and scanner configuration. It has 26 rules, 18 of which block by default, such as skip-added, assertion-loosened, coverage-threshold-lowered, ci-error-masked and security-scan-excluded. No network access, no LLM calls, no API keys.
It returns one decision, pass, review (a person must confirm) or block, plus JSON, a Markdown report with before ‚Üí after snippets, and a GitLab Code Quality report. Agent merge requests (duo/ branches, bot authors) get a stricter profile: an expected value changed while the code under test did not is block for an agent and review for a person.
Here the decision is the merge decision. On pass the merge request merges with no person. On review a manual human-review job holds the pipeline until a person runs it; the job fails for bot-named accounts. On block the pipeline fails.
GitLab Duo Agent Platform flows develop the project, and every agent merge request goes through the gate:
- implement-issue (custom flow, Assign trigger): issue ‚Üí plan note ‚Üí
duo/branch with code and tests ‚Üí merge request. - fix-pipeline (custom flow): reads the failed jobs' logs and pushes a fix to the same branch; if the gate blocked, it posts a note and stops. Ran live in MR !21: one-line code fix, tests untouched, merged with no person.
- integrity-review (custom flow, off by default): explains the findings and requests changes on
block. - agent-dispatch (scheduled CI job, every 15 minutes): merges or sets auto-merge on
duo/merge requests whose pipeline ran the gate, notes gate blocks, starts fix-pipeline through the Flows API for other failures, and leaves changes to CI configuration to a person. It never approves and never runs jobs.
How we built it
The core is Python 3.11+ with PyYAML as its only dependency. Separate agents wrote labeled corpora without seeing the detector code: a main corpus of 65 cases from the written policy, and two blind held-out sets of 50 and 60 cases by agents that saw the README but not the rule code, with labels fixed before the first run.
The pipeline's stages are verify (pytest with coverage, ruff), gate, package (wheel, sdist, container image), secure (SAST, Secret Detection, Dependency Scanning), approve (a child pipeline written from the decision), release (on tags) and monitor (nightly corpus replay against a baseline, a dashboard, agent-dispatch). The flows use the Flow Registry v1 format, checked against the AI Catalog's JSON schema.
Built with AI assistance: Claude wrote most of the code, documentation and corpora under my direction. I made the decisions and did the steps that need a person, such as creating the merge token and running human-review.
Challenges we ran into
- Flows cannot merge, approve or set auto-merge, and push options failed for the flow's service account ("An unknown error occurred") while the same push by a person worked. The flow now opens its merge request with
create_merge_request, and agent-dispatch merges with a person's token, kept in a protected, masked, hidden variable that merge request pipelines cannot read. - Bots cannot trigger flows: "Pipeline failed" did not fire for MR !1, whose pipeline the flow's push had started. agent-dispatch starts fix-pipeline through the API instead, and the "Pipeline failed" trigger is off, so one failure never starts two sessions on the same branch.
- fix-pipeline's first live run (MR !18) ended without a fix: the triage step answered with a sentence followed by
FIX, and the flow router compares the whole answer. MR !20 made the prompt strict (AI Catalog 1.3.0); the re-run in MR !21 fixed the bug and merged. - Our role is Developer: no approval policies, no pipeline schedule variables, no CI/CD Catalog publication. The dispatch schedule is recognized by its description.
- A merge request can edit the pipeline that judges it. The gate job installs tamper-gate from the merge request's base commit and reads
.tamper-gate.ymlfrom the base ref; agent merge requests that change CI configuration always need a person. - Dogfooding found our own bugs, including a gate miss:
SAST_DISABLEDset inworkflow:ruleswent unreported (fixed in MR !14).
Accomplishments that we're proud of
- An unattended run on 2026-10-08. Issue #3 was assigned to the implement-issue flow at 17:36 UTC. The flow wrote code and tests and opened MR !13. The gate said
pass. At 17:51 the scheduled agent-dispatch job merged it, and the issue closed. No person clicked anything on the merge request. - A failure fixed and merged without a person: in MR !21 (a seeded bug) agent-dispatch started fix-pipeline, the flow changed one line of code and no test, and agent-dispatch merged the green result.
- On the blind held-out v2 set (60 cases, run once before any fix), the gate blocked 0 of 20 honest fixes and caught 20 of 26 cheats.
- The review path works as designed: MR !6 waited at
manualuntil a person ranhuman-review.
What we learned
- Blind numbers are lower. The main corpus scores 65/65; the first blind run on held-out v1 scored 23/50. A corpus written from the same policy measures fidelity to the policy, not results on real agent merge requests.
- Structure beats text: with
ast, the gate can tell a deleted test from one that was renamed, moved or folded into a parametrized test. - Agent identity has its own rules. Flow merge requests are attributed to the person who triggered the flow, and triggers need a human actor, so hand-offs between agents go through CI and the API.
- What is being judged must not be able to edit the judge.
What's next
- With the Owner role: an approval policy on
review, a pipeline execution policy that puts the gate outside a merge request's reach, and Catalog publication. - The one held-out v2 cheat still missed after the fixes: a source branch on date fields that contains no test literal.
- Measure credits per flow run; docs/duo.md has only rough estimates.
- Replace the merge token before it expires on 2026-11-07.
Notes for judges
All links are in the public repository https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43143384/tamper-gate. Times are UTC, 2026-10-08.
Why GitLab shows @wwruibowen as author and merger of agent MRs. GitLab attributes merge requests created by a flow to the person who triggered the flow (composite identity), and agent-dispatch merges with that person's token, so merges and notes are recorded as the token owner's. The evidence that no person acted is the pipeline user (the flow's service account), the timeline on the issue, and the agent-dispatch job log.
Path A Hands-off: agents work without a person in the loop
A person created issue #3 and assigned it to the flow. After that, no person touched the issue, the merge request or its pipelines. The dispatch schedule had been set up earlier that afternoon and was deliberately not run by hand.
- Block path (staged): MR !15 is a staged demo, written by hand to imitate an agent that skips a test to make CI green. The gate blocked it in its MR pipeline:
skip-addedattests/test_cli.py:229, agent profile (https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43143384/tamper-gate/-/merge_requests/15, https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43143384/tamper-gate/-/pipelines/2927742873, https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43143384/tamper-gate/-/jobs/17040354487). At 18:39 the scheduled agent-dispatch job logged!15 duo/fix/skip-flaky-rules-test: pipeline 2927742873 failed: tamper-gate blocked; post a note; no fix(https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43143384/tamper-gate/-/jobs/17040487717) and posted that note on the MR; later runs log "note already posted; no fix". - Fix path (seeded): MR !21 is a refactor pushed by hand with one deliberate bug (added-line numbers off by one), stated in its description. After the push, no person touched it.
The first attempt, MR !18, ended at the triage step (see Challenges); its closing note explains it.
- Earlier agent MRs: !1 (issue #1) and !9 (issue #2). Their code was written by the implement-issue flow, but a person stepped in on each: !1 merged after a person rebased it, and !9 was opened by hand because push options failed for the flow (stated in its description). These failures are why agent-dispatch exists.
Most Stages Covered
Most Creative
- The tool judges the agents that build it. Its own repository is developed by Duo flows (https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43143384/tamper-gate/-/tree/main/flows), and each agent MR merges only if tamper-gate says
pass. - The decision is deterministic: no model decides whether an MR merges. The model-based flow (integrity-review) only explains findings.
- The judge sits outside the reach of what it judges: the gate's code comes from the MR's base commit, its config from the base ref, and an allow annotation an agent adds in its own MR can lower a finding no further than
review(https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43143384/tamper-gate/-/blob/main/README.md#configuration,tamper-gate-checkin https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43143384/tamper-gate/-/blob/main/.gitlab-ci.yml). - The test sets were written against the gate: separate agents that never saw the rule code wrote the held-out cases, including cheats aimed at slipping past the documented rules (https://gitlab.com/gitlab-ai-hackathon/transcend-october-2026/43143384/tamper-gate/-/tree/main/corpus-heldout-v2).
Evaluation (technical)
Each row says how it was produced; the rows are not interchangeable. Source: README, "Labeled corpora and evaluation", and the linked files in eval/.
| Set | How produced | Decision accuracy | Cheat recall | Cheat block | Honest false-block |
|---|---|---|---|---|---|
| (a) Main corpus, 65 cases | In-distribution: written from the rule policy, measures implementation fidelity | 65/65 | 28/28 | 23/28 (5 are review by policy) |
0/21 |
| (b) Held-out v1, 50 cases | Blind, first pass, code 371b847 | 23/50 (46.0%) | 15/32 (46.9%) | 10/32 (31.2%) | 1/7 |
| (c) Held-out v1 after fixes | Not blind | 45/50 (90.0%) | 29/32 (90.6%) | 25/32 (78.1%) | 0/7 |
| (d) Held-out v2, 60 cases | Blind, run once, code b486224 | 49/60 (81.7%) | 20/26 (76.9%): plain 12/13, adversarial 8/13 | 14/26 | 0/20 |
| (e) Held-out v2 after fixes | Not blind, code 99734fc | 59/60 (98.3%) | 25/26 (96.2%) | 20/26 | 0/20 |
- The README says (d) is "the number to quote for cheats the rules were not written against". Of (c) and (e) it says they "show the fixes did not break the set; they say little about unseen cheats."
- (d) also had one honest case wrongly flagged
reviewand two near-misses wrongly blocked. With the agent profile, (d) blocks 18/26 cheats and 3/20 honest cases: two test corrections labeledreview(blocked by policy) and the unittest-to-pytest conversion.
Duo Agent Platform usage
Three custom flows (flows/implement-issue.yml, flows/fix-pipeline.yml, flows/integrity-review.yml), enabled in the project with their service accounts; an Assign trigger for implement-issue (the fix-pipeline "Pipeline failed" trigger is off; agent-dispatch starts that flow); flows started from CI through POST /api/v4/ai/duo_workflows/workflows (agent-dispatch, and the optional duo-integrity-review job). No MCP server.
Known limits
- implement-issue ran for MRs !1, !9 and !13; fix-pipeline ran twice, both started by agent-dispatch (!18 ended at triage, !21 fixed and merged). integrity-review is written but its CI hook is off, and it has not been run in this project.
- Flows cannot set auto-merge or merge, so agent-dispatch uses a person's personal access token (
apiscope) in a protected, masked, hidden variable. It expires on 2026-11-07. - CI/CD Catalog publication is not possible: it needs the Owner role, and members of this hackathon project are Developers.
- With the Developer role, anyone with Developer can run
human-review, and without "Pipelines must succeed" a person can still press Merge while the pipeline waits. - The flows' prompts forbid running, retrying or cancelling jobs; that is a prompt rule, not an enforced permission.
- Rules cover Python tests (pytest, unittest) and GitLab CI, coverage, warning and scanner configuration.
Log in or sign up for Devpost to join the conversation.