Second Look

A GitLab Duo flow that checks every failed pipeline first: retry, config, dependency or a real code fix. On 184 real red pipelines it opened 12 needless fix MRs instead of 42.

Comment

Inspiration

When a pipeline goes red, AI auto-fixers answer with a fix merge request. But on real projects many red pipelines are not code bugs at all: the runner died, the network timed out, a registry was down, or the CI config is wrong. A fix MR there is noise a maintainer has to read and close. I wanted an agent that takes a second look first and decides what kind of failure it is before anyone writes a fix.

What it does

On every failed pipeline, Second Look reads the failed jobs, their logs, GitLab's failure_reason and the same job's history on that branch, and sorts the failure into one of four buckets:

  • no-code: runner, network or registry trouble -> retry
  • ci-config: the pipeline definition is wrong -> config change
  • dependency: an upstream package or image broke -> pin or bump
  • code: a real break -> goes to a human

A second agent records the decision as one issue per broken job, with the evidence (repeat failures add a note instead of a new issue). A maintainer-owned policy.json decides what is allowed (for no-code: retry once, never open an MR). Second Look never opens merge requests and never edits code (Supervised path).

How we built it

  • GitLab Duo Agent Platform: a custom flow (.gitlab/duo/flows/second-look.yaml) with two agents (triage + respond) and a Pipeline events -> Failed trigger.
  • Plain-code triage (agent/triage.py, rules in agent/evidence.py): failure reasons, log signatures and retry history. The decision is one function, used by the Duo flow, by the pipeline's own second-look job and by the replay eval, so the number below is the code you get.
  • Harvester + replay eval: collected real red-to-green streaks from public GitLab projects and replays them against the triage on every pipeline (eval/results.md is regenerated each run).
  • Python, GitLab CI/CD, GitLab REST API.

Results (honest)

184 real red streaks from gitlab-org/gitlab-runner, gitlab-org/cli, fdroid/fdroidserver and inkscape/inkscape. On the 94 where the maintainers' action is clear:

Same action as the maintainers Fix MRs opened where maintainers changed nothing
Always open a fix MR (auto-fixer) 38% 42 of 42
Second Look 62% 12 of 42

It still sends 9 of 36 real breaks the wrong way (to "retry"), and that is in the results table too. The other 90 streaks are excluded because the next green pipeline carried unrelated commits, so the true fix can't be known; counting them would make always-fix look 78% right, which is wrong.

Challenges we ran into

Ground truth. A first version counted any later green commit as "the fix"; it turned out 71 of 136 "code" labels came from unrelated commits (docs, translations, version bumps). We rebuilt the labels and only report the ones we can defend.

Accomplishments that we're proud of

An honest number on real data instead of a demo-only claim, and an agent that is useful precisely because it does less: it stops needless fix MRs and leaves code changes to humans.

What we learned

Most of the value is in the boring evidence (GitLab's failure reason plus retry history), not in the model. Code decides, the agent explains and records.

What's next for Second Look

Learn per-project retry thresholds from history, and hand confirmed code breaks to GitLab's Fix CI/CD Pipeline flow with the evidence attached.for Second Look

Built With

  • gitlab-api
  • gitlab-ci
  • gitlab-duo-agent-platform
  • python
Share this project:

Updates

Submission history