Pipeline Doctor

Inspiration

A red pipeline pulls someone out of their work: open the log, find the one line that matters, make a small fix, push, wait again. Most failures are routine: a lint error, an off-by-one, a bad version pin, a broken Dockerfile. "Life After Code" pointed us at that loop. We wanted an agent that handles it on the merge request where it happens, while a human still makes the final call.

What it does

Pipeline Doctor is a custom GitLab Duo Agent Platform flow, started by the native Pipeline events > Failed trigger (or on demand with a Mention). Five agents run in a chain:

1. Diagnose: reads the failing jobs, their logs, and the files the errors point to.
2. Classify: picks one category: lint, test, dependency, build, deploy, flaky or unfixable.
3. Fix / Retry / Report:
   • Fix edits the code, runs ruff and pytest locally (up to 3 attempts), and pushes a [doctor] commit to the same MR.
   • Retry reruns CI for flaky failures.
   • Report explains what a human must do.
4. Report: posts one structured note with the category, root cause, action taken, verification result, and a session link.

On main it never pushes. It opens a hotfix MR, or an incident issue when the cause is outside the repo (IAM, secrets, quota). It never merges: a human does. That is the Supervised gate.

Benchmark: 11 seeded failures, no human edits.

• 10/10 classified correctly
• 9/9 fixable failures fixed on the same MR
• 1/1 unfixable failure handed to a human
• Median 1.1 min to the fix commit and 6.0 min to green

How we built it

• Flow: .gitlab/duo/flows/pipeline-doctor.yml, five AgentComponents with a router. Each step gets a minimal toolset: only fix and retry can run commands, and only report can comment.
• Rules: AGENTS.md holds the taxonomy, commit conventions, protected files and verification commands.
• CI/CD:
  • lint, test, SAST, secret detection and dependency scanning
  • Cloud Build into Artifact Registry
  • Cloud Run in low-carbon europe-north1: no-traffic deploy, /health smoke test, then promote or roll back automatically
  • MR preview environments
  • a GitLab Release for every green main
• Auth: keyless, using GitLab OIDC and Workload Identity Federation.
• Carbon: every pipeline publishes an SCI estimate:

$$\text{SCI} = \frac{E \cdot I}{R}, \qquad E = \text{runner minutes} \times P_{\text{runner}}$$

where $I$ is the region's grid intensity and $R$ is one pipeline. For example: $0.0647\ \text{kWh} \times 30\ \text{gCO}_2\text{e/kWh} \approx 1.94\ \text{gCO}_2\text{e}$ per pipeline.

• Built with Claude Code subagents working in parallel git worktrees.

Challenges

• Strict flow schema. Every input_schema field is required, and several documented fields were rejected. We ended up with a minimal definition.
• Mentions from CI don't trigger flows, so we switched to the native pipeline-failure trigger.
• The Doctor couldn't start on the bug it had to fix. Its setup installed the branch's broken pins. It now falls back to a known-good toolchain.
• Previews leaked into production. gcloud run deploy inherits revision settings, so a preview's secret reached main. We added a separate preview service and --clear-secrets. The Doctor correctly refused to patch code for this and handed it to a human.
• Stale revision tags blocked every deploy. Each deploy now clears them first.
• No project tokens, so no auto-merge. That pushed us to a cleaner Supervised model.

What we learned

• Narrow scope builds trust. Small toolsets, a fixed classification step and one note per run keep the agent predictable and easy to review.
• Verify before push is what turns a suggestion into a fix you can merge.
• Saying "this is unfixable, here is what a human must do" is a feature, not a failure.
• Supervised doesn't mean slow. The human approves one verified commit instead of debugging.

Built With

• fastapi
• gitlab
• gitlab-duo
• google-cloud
• python