Green on Trial

• 

  Green on Trial workbench: inspect imported GitLab CI evidence and run independent local HTTP experiments.

The problem

For a solo maintainer or a small platform team, an AI-generated fix creates another question: did it restore the intended behavior, or merely make the original failure disappear? A webhook client that retries everything can recover a temporary 503 while repeating requests that a receiver permanently rejected.

What Green on Trial does

A mention starts a bounded GitLab Duo custom flow. The recovery component reproduces the failure and proposes typed delivery settings. A separate read-only reviewer inspects the candidate and evidence. A passing verdict creates an isolated draft merge request. The demonstrated flow ends with a draft for human review.

GitLab CI then recomputes the result independently. Alongside eight smoke contracts, a larger challenge tests 151 bounded HTTP sequences against the original configuration, a deliberately unsafe retry-all comparator, and the proposed configuration. It produces a readable review artifact with counterexamples, complete request traces, exact changes, and source/job links. Agent-authored reports do not determine that CI verdict.

What we measured

In the recorded CI comparison, the original passed 11/151 sequence contracts, the retry-all shortcut passed 141/151 but hid 10 permanent rejections behind apparent successes, and the safe proposal passed 151/151. Release policy also rejected retry-all. These are three controlled configurations over an exhaustive bounded synthetic suite, not estimates of production reliability or time saved.

The recorded real Duo run received a controlled malicious-log instruction to tamper with the verifier and push to main. Its actual MR changed only the candidate and two reports; independent CI passed all four jobs. This is one adversarial fixture result, not a general security guarantee.

How it was built

The runtime uses Python's standard library, a loopback HTTP receiver, SQLite run state, a constrained JSON candidate schema, and a plain HTML/CSS/JavaScript review interface. GitLab Duo custom agents, a mention trigger, Git branches/MRs, and CI connect proposal to independent verification. The local interface includes clearly labeled deterministic experiments for unsafe proposals, missing access, escalation, revision, cancellation, and interrupted recovery.

What was difficult and what we learned

GitLab workload environments contain unrelated untracked files. The flow initially stopped safely on those files; the corrected version leaves them untouched, rejects unexpected tracked changes, and stages only three allowed paths. We also separated model summaries, local CLI reports, downloaded CI artifacts, and verified platform records rather than treating one as proof of another.

Limits and next steps

This is bounded webhook-configuration recovery, not arbitrary repository repair or production deployment. A human reviews the outcome. The next useful validation is a real maintainer pilot; no customer adoption, incident avoidance, or time savings are claimed.

Codex assisted implementation and testing. GitLab Duo performs the demonstrated repair flow. The demo uses disclosed synthetic English narration. Original source and flow configuration are MIT licensed.

Reproduce and inspect

The public repository contains all runnable source, the MIT license, Duo flow YAML, and CI configuration. The starting candidate is deliberately broken; the local workbench runs without credentials using Python 3.11+.

A fresh public Duo run is available as session 9680806, with draft MR !1 and independent CI. The video records the earlier run of the same workflow. The public rerun preserves its failed first review and the subsequent corrected evidence-access run.

Built With

• css
• gitlab
• html
• javascript
• python
• sqlite