CI Failure Triage

• 

  Recorded GitLab Duo run: evidence collection, repair, verification and linked outputs. Earlier revision.

• 

  One-line application repair; five original tests pass in GitLab. Human approval and merge are still required.

• 

  Missing reports stay unknown, with an owner and next action. Earlier Duo run; wording later refreshed by the operator.

• 

  Same errors, different execution context. Live CI + operator check, not a new Duo run; no automatic reuse of an old fix.

What it does

A failed CI job is not yet a diagnosis. CI Failure Triage gives a GitLab maintainer an evidence-backed owner handoff or a tested repair merge request, while leaving the merge decision to a person.

The maintainer mentions the installed custom Duo flow with a failed-pipeline URL. The tool collects repeated job attempts and JUnit reports. GitLab Duo investigates the application code only when the evidence permits a repair attempt. It can change one allowlisted application file, run the unchanged tests and publish a draft MR. A separate check verifies the actual GitLab pipeline before marking that MR ready for human review.

If a test fails and then passes at one commit, the result is an intermittent observation, not a proven flaky test. If reports are missing, the flow stops with an assigned issue, the evidence gap and a next action. Repeat handoffs preserve maintainer notes outside the generated report. An explicit recheck can withdraw readiness when the repair head is no longer verified. This is not background monitoring.

Why I chose this problem

I chose the work around CI repair: what evidence survives, who owns the next step, and what a maintainer can actually approve. I did not want to treat a retry pass as a diagnosis or a model's final message as proof that a repair worked.

I also tested GitLab's native Fix Pipeline on the same application defect. It repaired it too, and it declined an unsupported repair on the same missing-report case. The prototype's additional output is a persistent owner issue with repeated-test evidence and explicit verification state. Whether that separate work item saves human effort remains unproven.

How I built it

This is a new, event-specific delivery-quote application, CI configuration and custom GitLab Duo flow for Start Fresh / Supervised. The application is a controlled workload, not a production service.

GitLab Duo performs runtime investigation and repair. GitLab hosts the CI executions, artifacts, issues and merge requests. Python's standard library handles API requests, JUnit parsing and deterministic checks. Publication checks the exact failed commit, allowed file changes and original test inventory. Readiness is tied to a returned GitLab commit and pipeline, not the agent's confidence.

What the runs show

• An earlier Duo run repaired one threshold comparison. The unchanged five application tests and 42 tool tests passed in GitLab. The MR remains unmerged.
• Missing reports stayed unknown and produced an owner handoff, not a repair MR.
• An explicit Duo recheck restored Draft after an operator deliberately regressed a verified repair. Duo made no code changes in that recheck.
• Current-code CI passed 116 tool tests and five application tests. A newer history comparison found matching errors but different Python environments and kept that difference visible. This latest check ran through the operator API path against real CI artifacts; the updated Duo flow is installed but has not yet been rerun.

A challenge and what I learned

Repeating a handoff could overwrite an investigation note in the issue description. I changed the tool to replace only its intact generated block and preserve surrounding maintainer text. Edited or ambiguous blocks now stop for owner review. Regression tests cover this behavior, and a live operator check read back two issues after repeat handoffs and confirmed that their saved note regions stayed unchanged.

The lesson was that a useful handoff must survive the next run, not just look complete once. There are still limits: concurrent edits are not transactionally locked, history lookup is bounded, missing old context stays unknown, and a historical match does not authorize copying an old patch.

Limits and attribution

These are synthetic development cases, not production incidents, an independent accuracy benchmark or a human-efficiency study. No adoption, time savings or superiority over native GitLab is claimed. The checks constrain the application path; they are not a complete security boundary around an agent with shell and OAuth access. The prototype stays in a controlled project and never merges, quarantines tests or changes permissions.

Original project code and YAML are MIT licensed. Development and submission preparation were assisted by OpenAI Codex; GitLab Duo performs the demonstrated runtime work. The validation record preserves the versions, failed attempts, native comparison and evidence limits.

Built With

• gitlab
• python
• yaml