AfterGreen — Verified CI Repairs

Inspiration

A green CI pipeline is a useful signal, but it does not always mean an AI-assisted repair is correct.

An AI coding agent might make a failing pipeline pass by weakening a test rather than fixing the underlying problem.

We built AfterGreen to investigate a simple question: How can we evaluate the evidence behind an AI-assisted repair instead of trusting the green pipeline alone?

What it does

AfterGreen is an evidence-driven verification prototype for AI-assisted software changes in GitLab CI/CD.

It evaluates supplied test results, changed file paths, and explicit verification policies to distinguish a passing candidate from a verified repair.

Our controlled research covers four scenarios:

• A baseline defect.
• An honest application fix.
• A weakened test that conceals broken behavior.
• A legitimate test change.

In one controlled local report, the candidate test suite exited successfully, while external behavior checks passed only one out of five cases.

AfterGreen makes this distinction visible rather than treating a successful test exit as sufficient proof of correctness.

How we built it

We developed AfterGreen using Python, deterministic verification logic, structured evidence reports, automated tests, and GitLab CI/CD.

The prototype includes:

• Controlled research fixtures and independent behavior checks.
• Evidence evaluation and policy-based review.
• Structured reports with content hashes.
• A visual evidence-review interface.
• A GitLab Duo-assisted experimental workflow.

In our GitLab Duo experiment, a read-only agent prepares a structured JSON proposal. A human input step precedes a deterministic checkout diagnostic.

The experiment does not automatically merge or repair source code.

We also implemented checkout identity checks to investigate whether the code being reviewed matches the intended source commit.

Challenges we ran into

One major challenge was separating successful execution from trustworthy verification.

In our first recorded GitLab Duo experiment, checkout identities differed across the human input boundary. The diagnostic exited with code 23, even though the outer CI job passed.

A follow-up experiment fetched the pinned source commit and produced a diagnostic exit code of zero.

However, we deliberately reported the matching checkout as unattested rather than fully verified. Successful execution alone cannot establish the authenticity of all evidence or independently prove human approval.

This distinction became a central part of our verification design.

Accomplishments that we're proud of

• Built a working AfterGreen verification prototype and evidence-review interface.
• Implemented four controlled research scenarios.
• Demonstrated a case where candidate tests passed but external behavior checks passed only one out of five cases.
• Integrated a read-only GitLab Duo experimental workflow.
• Detected a checkout identity mismatch that was not reflected in the outer CI job's passing status.
• Passed 122 local unit tests.
• Passed all four configured GitLab CI jobs in the hosted pipeline.
• Merged the reviewed implementation into the main branch.

Our results demonstrate the importance of preserving evidence and uncertainty in automated verification.

What we learned

We learned that a passing pipeline, a successful diagnostic command, and a verified repair are three different things.

We also learned that deterministic checks are most useful when their limitations are explicit.

A report hash identifies reviewed content, but does not authenticate its original source. A matching checkout does not independently prove human approval. Missing evidence should remain visible rather than being silently treated as success.

What's next for AfterGreen

Our next steps are to strengthen evidence provenance, extend controlled repair evaluations, improve GitLab Duo integration, and test the verification workflow against more realistic AI-assisted changes.

We also want to expand the evidence-review interface and investigate how teams can use deterministic verification without slowing down development.

Our principle is simple: A green pipeline is a signal, not a verdict.

Built With

• gitlab
• gitlab-ci/cd
• gitlab-duo-agent-platform
• pytest
• python
• yaml