polyman: Who judges the judge?

The whole loop: every box is a person, a Duo flow or a CI job on main. CI fails an agent push to an MR branch or main without a go-ahead.
Live on Google Cloud Run: polyman 3.1.0 checks a sample problem and finds the planted mistake.
18 Oct: polyman 3.0.0, as installed from npm, matched 0 of our 5 test problems; the fixed 3.0.1 matched 5 of 5.
#22: issue-builder proposes a fix and stops; a teammate who didn't file the issue gives the go-ahead.
!67: our review agent was told to report only and pushed anyway; the CI guard failed the push for having no go-ahead.

Path B · Assisted · all nine DevSecOps stages. Special prizes: Most Stages Covered, Most Environmentally Impactful. Google Cloud: Cloud Run.

Who judges the judge? polyman checks programming-contest problems before a contest uses them. On GitLab, four GitLab Duo Agent Platform flows propose and build fixes, people approve and ship them, and CI fails any agent push to an MR branch or main that lacks the required go-ahead. From 17 to 24 Oct (ledger recount, 25 Oct 01:00 UTC), all 57 merged agent loops and all 79 accepted agent pushes in the ledger had a go-ahead from someone other than the requester; separately, the guard failed 9 attempted agent pushes on 8 merge requests. Every release since 3.0.1, four with 3.1.0, was installed from npm and judged against the one before it before a person made it latest. We also measured our own review agent on five frozen merge requests: it finds 13 of 17 known P1/P2 findings in each of two runs, at about 8 credits a review. The ledger is public and needs no login.

Video, under 3 min · Original project (MIT): https://github.com/HamzaHassanain/polyman · Our project: GitLab · Live (Cloud Run, serving polyman 3.1.0): https://polyman-verify-ncvkutguda-ue.a.run.app · For judges: the 60-second path

polyman is Hamza Hassanain's MIT project, and Hamza is on this team. We imported its 3.0.0 release at dcb5779 (tagged hackathon-start) with its history intact, then added four Duo flows, the go-ahead checks, judging of each release as installed from npm, deployment checks and incident handling. Everything after that tag is hackathon work.

Before and after the move to GitLab (from the README's before/after table):

	Before (GitHub, polyman 3.0.0)	After (GitLab)
Is the verdict right?	unit tests only	real compilers on the corpus: blocking on every MR that changes polyman's code (since !250), on every `main` pipeline, and nightly on the published package
Push to Polygon	trust the ✓	upload to Polygon, then read it back and compare
Releases	no publish workflow	canary, npm `next` judged against `latest`, then a person promotes it, with provenance
A live service	none	Cloud Run candidate checked by a golden set, manual promote, automatic rollback
Who finds the bugs	users; two user-reported bugs waited 230 and 175 days for a release with the fix	the judges, reviews and audits: 18 issues labelled `found-by`
Who changes the code	people	four Duo agents propose and build; a different person approves each step and a person merges

The problem

A contest problem is a statement, tests, a validator, a checker and solutions that must each get the right verdict. If the tool that checks them is wrong, a broken problem looks fine to its author, and contestants find out mid-round. polyman's own template tells AI coding agents that polyman verify is "the bar for 'done'", yet in 3.0.0 its JSON marked tests the checker never judged as OK (#21). On 18 Oct the version people had installed matched 0 of our 5 corpus problems; 3.0.1, fixed through the loop below, matched 5 of 5 and passed a 34-check Polygon round trip (release check).

What judges polyman: programs, not a model

The corpus: five original problems with deliberately wrong solutions, run with real g++, Java and Python, every verdict checked per test, nightly against the version people install from npm. It caught the published 3.0.0 accepting wrong answers (#45).
Mutation testing of the tests: they catch 574 of 693 bugs planted in the verdict code, 82.8% (job, 25 Oct).
A CI guard on agent pushes to MR branches and main: autonomy:goahead matches each push to a go-ahead in GitLab's own note and push records and fails it if the go-ahead is missing or came from the requester; autonomy:guard fails any agent change to CI, the policy, the engine or the corpus. Its own tests catch 300 of 330 bugs planted in its rules, 91% (job).
A session audit with no model in it, which a person runs: it checks every push in each agent session's trace against GitLab's push events (#103).

One test went against us. On 18 Oct we committed the hash of an answer key, then ran seven seeded bugs and five harmless refactors past the corpus (#48, job). The unit tests caught 6 of the 7, the corpus 3, with no false alarms, and we did not change the patches or the scoring afterwards. The other side is real too: the 8 bugs in the published 3.0.0 passed 967 unit tests, and the judges found all 8 (4 by the corpus, 4 by the Polygon round trip). Neither layer is enough alone, so both run.

One change, end to end

Of the 57 loops a person merged, 37 were changes issue-builder built, 18 were reviewer fix rounds on people's own MRs, one was a pipeline fix and one an incident revert. The median from go-ahead to merge is 3 h 50 min (12 to 51 minutes for the first five). Example: Ahmed filed #95, issue-builder proposed, Mazen gave the go-ahead, the agent opened !186, CI matched the push to the go-ahead, and Hamza merged.

Step	Agent (Duo flow)	Person, in GitLab	Checked by
Ask		files an issue with "done when", assigns issue-builder
Propose	posts the change, the failing test and the risk tier; stops		the flow's rules
Go-ahead		a teammate who didn't file it replies `go-ahead`; at high risk, someone on our maintainers list (`.autonomy.yml`)	`autonomy:goahead`
Build and review	pushes once, opens the MR; mr-reviewer reviews at a depth set by risk	asks for each re-review; a go-ahead per fix round	`autonomy:goahead`, `autonomy:guard`
Merge and ship	names who may merge	merges; clicks the npm and production promote jobs	manual jobs; only the rollback is automatic

A refused push: a review agent told "REPORT ONLY: do not push" pushed anyway, and CI failed it (!67). A prompt isn't a control; the check is. In all, the guard failed 9 agent pushes on 8 MRs: 7 had no go-ahead (all on 18 Oct), 1 changed files only people may change (!177), and 1 was a merge commit with no proposal before it (!241).

The reviewer, measured

We tested our review agent on a frozen set: five old merge requests whose first reviews had 17 known P1/P2 findings, plus two held-out MRs it was never tuned on. We wrote the pass/fail bar down before each run. It was public before the run for E-R (!165) and E-R2 (!200); for the others, including the shipped E-B35, it was in our own notes, with a time we report ourselves. Every bar, verbatim: preregistrations.md.

Reviewer	Earlier P1/P2 findings found	Two held-out MRs	Credits per review (median)
Original full reviews	the reference (17)		13.1
1.0.32, two runs	7 of 17		5.3 to 6.7
Shipped (E-B35), two runs	13 of 17 each	4 of 5 findings (first run)	7.8 and 8.1

It raised 19 P1/P2 findings in one run and 22 in the other; 17 and 19 were real, with no false P1, but one false P2 in each run and one and two overrated findings respectively (including run 2's real bug rated P1 instead of P2). We dropped a version that also found 13 of 17 but raised 35, only 17 real, and one false P1. The shipped one costs about 40% less than the original full reviews, but it is not the same coverage: 4 of 17 are still missed or rated P3 in each run (SUSTAINABILITY.md).

Impact: the loop is not polyman-specific

The four flows, the go-ahead guard, the ledger and the per-MR carbon report read one policy file, .autonomy.yml, not polyman's code. The same loop ran end to end on a second project of ours, a small orders API: issue #3 → MR !20, merged 21 Oct, with its own .autonomy.yml. That is one loop on a project we also run, not outside adoption.

polyman is where we measured it. A contest-problem setter outside the team ran polyman 3.0.3 on his own Polygon problems, on his own machine (field report #106). On one tested problem (534794), polyman and Polygon agreed on all 22 tests, and polyman caught 3 of the 4 planted bugs that applied; the miss exposed a real gap in his own problem. His four findings were fixed in 3.1.0, three of them by agent loops: #108 (a missing validator is reported plainly), #110 (pull says which testsets it downloaded) and #111 (verify warns when a validator has no INVALID self-tests). On another problem, one verdict difference with Polygon is still open (#107). Before the move, two bugs users reported waited 230 and 175 days for a release with the fix; here the median from go-ahead to merge is 3 h 50 min, a different interval from report to release. npm recorded 2,194 downloads from 7 Nov 2025 to 24 Oct 2026: downloads, not people.

Nine DevSecOps stages, each with automation and a human gate

Every stage with a run you can open, and one bug (#21) followed through all nine, from the nightly that caught it to the release people install, with each job linked: Stages.

Stage	What the automation does	What a person approves	Evidence
Plan	issue-builder proposes and waits; issue templates; a judge dashboard	the go-ahead, from someone who didn't file it	#22 proposal
Create	issue-builder opens the MR; mr-reviewer reviews at a depth set by risk	a go-ahead per fix round; a person merges	#22 to !26
Verify	the corpus, mutation testing, the nightly run on `polyman-cli@latest`; pipeline-fixer proposes on red	the fixer pushes once, on a go-ahead	2,756 tests; also on a Windows runner (2,694 passed, 62 skipped) and the corpus on macOS (5 of 5)
Package	GitLab's registries first: canaries in this project's npm package registry, the verify service image in its container registry, each release's evidence in its generic package registry; then npm `next` with provenance	a person clicks canary and npm-next	packages, container registry, 3.0.1 publish
Secure	SAST, secret and dependency scanning feed the risk score; SBOMs	a person triages the findings, and gives the go-ahead on the fix	#20: 349 triaged, 1 real; issue-builder built its fix (#62 → !136) on a go-ahead
Release	`release:next-check` installs both versions from npm and judges them before `latest` moves; a GitLab Release with evidence; Cloud Run candidate at 0% traffic, then promote	a person clicks promote	3.1.0: 5 of 5, round trip 34 of 34, no regression
Configure	OpenTofu with GitLab-managed state; nightly plan checks drift and policy	a person merges any infrastructure change	infra:plan: no drift
Monitor	a production check reads the review's watch list; automatic rollback; incident-responder opens an Incident and a revert MR	a go-ahead from our maintainers list on the revert	drill: #65, about 94 s of bad production
Govern	GitLab's Release Evidence on every release and vulnerability management: on 18 Oct a person triaged all 349 findings then in the vulnerability report by hand, recorded on #20 (our role can't change a finding's status); the go-ahead guard, protected paths, a SHA-256 ledger	at high risk, our maintainers list, never the requester	v3.1.0 release, triage on #20

What the agents cost, and what we cut

We estimate the model calls' footprint from the checkpoints of 949 recorded Duo sessions (16 Oct to 25 Oct 00:32 UTC), using SCI (ISO/IEC 21031:2024) and the Green Software Foundation's SCI for AI; in CI, the GSF Impact Framework (if-run 1.1.1) recomputes every review's SCI and the job fails if the two disagree. These are estimates, not measured model energy. Model calls are 76 to 97% of a high-risk review's footprint, so we cut there first, then CI, then the product.

Layer	What changed	Measured	How
Agents	review depth follows risk; fewer wasted sessions	wasted reviewer credits: 31% before reviewer 1.0.17 (16–18 Oct), 8% afterwards to 19 Oct 20:03 UTC, excluding 14 short stops; low-risk reviews use about 1/5 of high-risk reviews' calls (medians through 25 Oct 00:32 UTC)	session checkpoints
Agents	the shipped reviewer (E-B35)	about 40% below the original reviews' 13.1 credits, at 13 of 17	two runs, same 5 MRs
Agents	tried and reverted: a cheaper model, a "reproduce every finding" rule, a shortcut read	the cheaper model looped and cost more; the rule missed its bar (9 of 17 against 15); the read failed in real sessions	bar written before each run (see above)
CI	the mutation job reuses the unmutated build's binaries	mutation-survivors runner-minutes per pipeline, median 185.1 to 65.2 (-64.8%); all 116 mutant outcomes identical in the 3 safety-check pipelines	30 against 8 pipelines
CI	each pipeline reports its own footprint in its MR	!330: 3.58 g and 28.4 runner-minutes, against 15.91 g and 126.2 on `main`; the two run different jobs, so this is a measurement, not a saving	a `.post` job (!252)
Service	testlib and the C++ header built once per image	-57.4% CPU per verify, 90 of 90 outputs byte-identical, candidate latency 18.3 to 9.4 s	local CI-built images, paired n=45 (acceptance note); latency is a separate Cloud Run sample, n=5 per revision (SUSTAINABILITY.md)
CLI, on a setter's machine	a precompiled `<bits/stdc++.h>` per compiler and flags, shipped in 3.1.0	-28% CPU per verify of a fresh problem copy, header already cached (n=15, 95% CI -33% to -24%); -18% per edit-and-reverify (n=15, -21% to -15%); identical verdicts, tests, outputs and binaries in all 65 runs. First use, empty cache: about -15% CPU (n=5), no wall-time gain shown	3.1.0 from npm, PCH on against `POLYMAN_PCH=0`, one Linux laptop, RAM-backed storage in both arms, 25 Oct; clang skips it

The CPU saving again, with its bar public first: on 26 Oct we posted the pass bar on #113 before the run started, then re-ran the released 3.1.0, PCH on against POLYMAN_PCH=0, on SSD storage. It passed (result, raw data and script in !362): -25% CPU per verify of a fresh problem copy, header already built (n=15, 95% CI -30% to -19%) and -16% per edit-and-reverify (n=15, -20% to -13%), with identical verdicts, tests, outputs and binaries in all 60 runs.

For the 57 ledger loops merged 17–24 Oct, with sessions counted through 25 Oct 00:32 UTC, the median estimated footprint of proposal, build and review sessions is 12 to 98 gCO2e (about 38 g central), totalling 825 to 6,828 g. This excludes the changes' own CI pipelines, the verify service and model-hardware embodied emissions. The model-energy assumption spans about tenfold, because nobody has measured our model's energy: we use 0.24 to 2.5 Wh per call and 579 gCO2e/kWh (us-east1). The warm verify instance adds 92 to 146 g a day, counted as a cost. How to recompute every figure: SUSTAINABILITY.md.

What we learned

A prompt isn't a control; the check is. A review agent told "REPORT ONLY: do not push" pushed anyway (!67). The guard caught it, not the prompt.
Measure the agent, not the prompt. Three reviewer changes that looked better were dropped when they missed a bar written before the run (E-R, E-R2, E-B34). The one we shipped found 13 of 17.
Neither layer is enough alone. Unit tests caught 6 of 7 seeded bugs and the corpus 3, but the corpus and the round trip found all 8 bugs in the published 3.0.0 that 967 unit tests missed.
Write the bar down where others can see it. Our early bars lived in our own notes. The last one (#113) was public before the run, and it passed.

Limits

Nobody on the team has GitLab's Maintainer role, so main's protection is the hackathon default: no force pushes, but Developers (our role) can push and merge. A push straight to main is checked after the fact: autonomy:guard:main fails that pipeline, but it can't undo the push, so people control merging. In 21 of the 57 loops (to 24 Oct) the person who gave the go-ahead also merged: allowed, but weaker separation.
GitLab's job token can't read agent sessions, and we chose not to store a personal token in CI, so a person runs the session audit; it can over-report pushes, so each report is checked against GitLab's push events.
The session audit also found two accepted agent pushes missing from the ledger after people rebased the branches (!308 and !321); the headline counts cover ledger entries.
The reviewer misses 4 of 17 earlier findings in each run; four known P2s are open on main.
3.1.0's confirming run passed on a retry: the first attempt couldn't install the version yet because npm hadn't made it visible; both attempts are in the same pipeline.
In 3.1.0, a solution tagged RE that never crashes still passes verify; the fix (!352) waits until after judging.
On Ubuntu with apport, an aborting solution can be judged TLE instead of RTE (sudo systemctl stop apport fixes it).
The verify service stays in us-east1 so its URL doesn't change; it stays live until winners are announced (about 16 Nov).

What's next

Move the judges and the guard back to polyman's home repository; a spend cap per loop (GitLab's job token can't read spend today); a security policy project for Govern; and docs-only changes graduating to Supervised under written criteria, once their record allows. The four flows are public in GitLab's AI Catalog, with a guide to adopting them.