Verdict Gate

AI agents propose fixes; deterministic verifiers decide if they merge. One GitLab Duo flow, one VERDICT format, pluggable checks for dependency upgrades, flaky-test fixes and translations.

Comment

Inspiration

Code generation is solved. What is not solved is trust. An AI agent opens a merge request that says "fix flaky test" or "upgrade express to 5", the pipeline turns green, and someone has to decide whether that green means anything. A test that fails 15% of the time passes three times in a row with a 61% probability. A release note tells you what changed for everyone, not which of your lines will break. We had lived this on our own projects: dependency upgrades postponed for months, flaky fixes that came back red weeks later, translations shipped with tags and placeholders silently lost.

We wanted one rule for the whole post-code lifecycle: the agent proposes, a deterministic verifier decides.

What it does

Verdict Gate is a merge-request gate with pluggable verifiers. Every verifier runs as a CI job (verdict-gate:verify), judges the MR against the base branch, and prints a single line to the job log:

VERDICT {"schema":"verdict-gate/1","verifier":"flaky","outcome":"PASS","detail":"FIXED: residual failure rate below 5% at 95% confidence","evidence":[{"where":"base","runs":100,"failures":17},{"where":"head","runs":100,"failures":0},{"p":3.63e-06,"alpha_used":0.0167},{"upper_bound_95":0.0295}],"guards":[],"needs_human":false}

Four outcomes, always the same: PASS, FAIL, UNSURE, BROKEN. The evidence is something a human can re-check in a minute: a file and line, a failure count, a p-value, an upper bound. The exit code of the job encodes the outcome (0, 2, 3, 5, and 6 for PASS with a guard), so the merge request pipeline is red whenever the gate is closed.

Five verifiers today, all behind the same contract:

  • flaky: reruns a test on both sides of the MR (100 runs each, alternating), applies a one-sided Fisher exact test and a 95% Clopper-Pearson upper bound, and says FIXED, NOT FIXED, NEED MORE RUNS, NOT REPRODUCED or BROKEN. A diff guard catches the classic cheats: removed or loosened assertions, added skips, added retries, raised timeouts. A FIXED with a guard becomes FIXED BUT TEST WEAKENED and requires a human. A bisect mode finds the commit that introduced the flakiness with the same statistics.
  • upgrade: compares two published versions of a dependency and crosses them with the exact lines of your code that use them: removed members, ESM-only packages you still require(), deep imports that no longer resolve, Node engine requirements.
  • i18n: checks a translated catalog against its source: missing keys, strings left in the source language, HTML tags and placeholders not preserved, line breaks, Unicode normalization, lengths that overflow.
  • perf: runs a benchmark command on both sides, Mann-Whitney U plus a bootstrap confidence interval on the median, and refuses to call a regression on noise.
  • api: diffs an OpenAPI contract between base and head and classifies every change as breaking, additive or cosmetic.

Then the GitLab Duo Agent Platform reports. A custom Duo agent and a custom Duo flow (two agents in a YAML file: one only reads the merge request, the failing jobs and the job log; one only writes the note) extract the VERDICT line and post one note on the MR with the evidence copied verbatim, saying whether a human is needed. The AI never changes the outcome or the numbers. An MCP server exposes the same verifiers as nine tools to any agent, including Duo Agentic Chat.

How we built it

  • Each verifier is a standalone, zero-config CLI with no API key and no account: Python with only the standard library for flaky, perf, api and i18n; Node with the TypeScript compiler as its only real dependency for upgrade.
  • A JSON Schema (schema/verdict-v1.json) defines the shared contract; a conformance test runs against every verifier's output in CI.
  • A CI/CD component (templates/verify.yml) selects the verifier from the MR label or the branch name, validates the verdict against the schema, prints the line and sets the exit code. A scheduled job (templates/monitor.yml) re-measures the quarantined test and the outdated dependencies on the main branch.
  • The custom Duo agent "Verdict Gate reporter" was created in the AI Catalog with the GitLab tools it needs (get_merge_request, get_pipeline_failing_jobs, get_job_logs, create_merge_request_note and a few more). The custom Duo flow (.gitlab/duo/flows/verdict-gate.yaml, flow registry v1, ambient environment) is enabled on the project with a trigger on pipeline events (passed, failed) and on mention of its service account.
  • The MCP server (mcp/verdict_gate_mcp.py) speaks stdio JSON-RPC 2.0 and is registered in .mcp.json.

What actually ran (25 October 2026, visible in the repository)

  • Four demo merge requests with the expected verdicts: !1 placebo sleep (FAIL, not fixed, p = 0.92), !2 real fix (PASS, 17 failures before, 0 after), !3 loosened assertion (PASS but test weakened, needs a human), !4 express 4 to 5 (BREAKS at src/server.js lines 9 and 10).
  • The custom Duo agent ran from Duo Chat on !3, asked for approval and posted the note.
  • The custom Duo flow ran twice on !2, once from the pipeline event after a push and once from a mention, and posted the verdict note by itself as its service account.
  • What is not done: the agent's fix-proposal commit was not exercised on !4; the i18n, perf and api verifiers run in CI but are not wired to the flow yet. The README has a "Status and honesty" section with the same list.

Challenges we ran into

  • Making a verifier that says "I don't know". NEED MORE RUNS prints the exact number of clean runs required (59 for a 5% residual rate, 299 for 1%).
  • Keeping the statistics honest with repeated looks: alpha is divided by the number of looks (Bonferroni), runs alternate base and head so machine noise hits both sides equally.
  • The first version of the upgrade engine flagged two packages as ESM-only that still ship a CommonJS build in a nested folder. We fixed the classifier to read the nearest package.json the way Node does, and pinned the case with a test.
  • On the Duo platform, a trigger cannot target a chat agent, so the autonomous part became a real flow with two components and explicit routers. Prompts with HTML comments are rejected, and the flow editor needs plain YAML without templating in the outputs.

Accomplishments that we're proud of

  • Exact numbers, not vibes: the same statistics that decide on the MR are unit tests.
  • A demo MR that loosens an assertion gets FIXED from the statistics alone and FIXED BUT TEST WEAKENED from the guard. That is the whole point of the gate.
  • The upgrade engine reads two real versions of a package without ever running its install scripts or calling its code.
  • A flow that posted the right evidence on its own, twice, with the numbers copied verbatim.

What we learned

A green pipeline is a claim, not a proof. The useful place for a language model is reading evidence and writing the report or the fix; the useless place is deciding whether something is broken. Once the decision is deterministic, autonomy becomes a policy switch rather than an act of faith.

What's next for Verdict Gate

  • Wire the i18n, perf and api verifiers to the flow and let the agent propose the fix commit on BREAKS verdicts.
  • The hands-off policy for teams that want a full loop on PASS verdicts without guards.
  • Sharding long measurements across CI jobs; more verifiers behind the same contract (schema migrations, accessibility of rendered pages).

Built With

Share this project:

Updates

Submission history