No Guesswork

Seven agent phases on GitLab Duo, seven gates. A separate judge in CI asks one question: can the next agent act on this without guessing?

Comment

Inspiration

I've spent years in test automation, and the failures that hurt most were never typos. They were gaps in the requirement that someone filled with a reasonable-looking guess. Coding agents make this worse: they can write the code, but they can't ask what you meant, so they guess, and you find out after the code exists.

So the question behind this project was: what if every hand-off between agents had to pass a gate that asks one thing, can the next agent act on this without guessing?

What it does

No Guesswork is a gated agentic pipeline on the GitLab Duo Agent Platform.

  1. Contract. A human writes the feature contract in the issue, interviewed by a concept-partner agent in Duo Chat. It reads the product docs first, then asks about limits, exact texts, error precedence, and contradictions with earlier decisions.
  2. Seven gated phases. A custom flow runs concept → spec → implementation → testing → security → release → validation. After each phase, a separate judge job in GitLab CI scores the artifact. Deterministic evidence comes first: failing tests or scanner findings become blocking issues and cap the score below the pass mark, whatever the model thinks.
  3. Human in the loop. On a FAIL the flow stops. I decide: continue, or reject: <phase> to send it back. I label gate verdicts (valid / false-pass), and the governance report tracks how often the judge agrees with me.
  4. Merge and deploy. When all seven gates pass, the flow opens a merge request. I merge it, a release is published, and CI deploys it to a Google Cloud VM, health-checks it, and a scheduled job monitors it every hour.

The pipeline built a real app: a sandbox for training small language models on CPU. Pick an example text, train, and compare two runs. One chart shows the difference between a model that memorized its training text and one that learned.

Results

From the governance report, built from the gate history in git:

  • 8 features (#4–#11), each through all seven gates: 56 phase gates.
  • 68% of phases passed on the first attempt (73% excluding failures caused by the pipeline itself).
  • 2 features passed all seven gates first time (#8, #10).
  • 22 failed attempts, each with a recorded cause: 6 my contract, 7 the agent, 9 the pipeline.
  • 9/9 GitLab lifecycle stages covered, from Plan to Govern. On #11, the release was live and health-checked 3 min 21 s after merge.

Two moments that show why the gates matter:

  • #9: all 398 tests passed, and CI was green. The judge still failed implementation at 5/10: evaluate() never restored training mode, so dropout would have been silently off. The retry passed at 9.
  • #7: my own contract drew the loss chart upside down, and the concept gate passed it. The testing gate exposed it 30 minutes later. I sent it back, the gate failed my fix twice before it passed, and I labelled the first pass false.

How we built it

  • GitLab Duo Agent Platform: a custom flow with 16 components (dispatcher, 7 phase agents, 7 gates, finalizer) and a concept-partner agent in the AI Catalog. The flow's service account wrote about three of every four commits on main.
  • Judge: a Python job in GitLab CI calling Claude. It runs outside the flow on purpose, so the agent that did the work doesn't grade it. Deterministic rules cap the score, a ratchet stops retries from scoring lower without reason, and an evidence summary with failing test names goes first in its input.
  • Governance: a script computes first-pass yield, attempts and interventions from .pipeline/ state, with every failed attempt annotated by cause (human:, agent:, pipeline:). The report is published to GitLab Pages.
  • Deploy: Docker on a Compute Engine VM behind Caddy with automatic HTTPS. The deploy job runs only on a push to main by the maintainer, and all deploy secrets are protected, masked and scoped to the production environment.

Challenges we ran into

  • Session failures without a cause. Some agent sessions stopped with no reason shown. We re-ran the same input several times before changing it and narrowing down the cause. On #11 it turned out to be the size of the issue description, so we moved long texts into files.
  • A planned stop looks like a crash. When a gate fails, the flow stops by design, but the session shows as "failed (dropped)". Telling a planned stop from a real failure took reading the comments.
  • CI clone timeouts. Occasional gateway timeouts during checkout; retries (GET_SOURCES_ATTEMPTS) fixed it.
  • Making the judge reliable. Early versions lost evidence to truncation and sometimes returned unparseable JSON. Putting an evidence summary first and parsing robustly fixed both.
  • Building the pipeline while using it. #4 ran while phases were still stubs, so most pipeline-caused failures are there, and they're annotated as such.

Platform feedback, with session IDs, is in docs/platform-feedback.md.

What we learned

  • The contract decides almost everything downstream. Most retries trace back to something the contract left open.
  • Tests aren't enough, and a judge alone isn't either. #9 had green CI and a real bug. #7 had a judge pass on a wrong contract. Deterministic checks first, a judge where only judgement works, a human where it matters.
  • Measure the gate, not just the code. Labelling verdicts made it visible when the judge was wrong, and that's what supervision means in practice.

What's next

More labelled verdicts to measure judge agreement properly, skipping judge runs on intermediate commits, and making the provider configurable in one place.

Built With

Share this project:

Updates

Submission history