Self-Healing CI/CD with Governance

The title + the “How it works” diagram
The top: “Powered by Duo Fix Pipeline flow”, plus the duo-flow and self-heal labels
“Self-heal governance passed” and “Pipeline passed with the Duo fix”
“Approved by praveendataguy” and “fix verified”
Human approval required before this patch runs: target ref main is production. Approve or reject in Slack.

Inspiration

AI agents can now fix a failing pipeline on their own. That raises a new question for every team: how do you trust a code change that no person wrote? An agent under pressure to make CI green might skip a test, touch a secret, or push straight to production. I wanted autonomy and control: let the AI fix things, but put a safety layer around it.

What it does

Self-Healing CI/CD with Governance uses two agents, each with one job:

The fixer: GitLab Duo's Fix CI/CD Pipeline flow. A pipeline failed trigger starts it automatically. It reads the job log, finds the root cause, edits the code and opens a merge request.
The guardian: my governance agent. It inspects every Duo fix MR and decides what happens next:

If Duo's fix...	Then
is a normal code change on a feature branch	Auto-verified: a test pipeline runs right away
targets `main` or a release branch	Human approval in Slack first (the MR is held as a Draft)
changes CI, Docker or infrastructure files	Human approval
skips or weakens tests to make CI green	Human approval
touches secrets or keys	Blocked

After verification it comments on the MR and posts to Slack: verified or needs a human. Every decision is written on the merge request, so the MR becomes the audit trail. Nothing is ever merged automatically.

Autonomy level: supervised. Automatic where it's safe, a human where it matters.

Proof it works

On the public demo project:

MR !4, the automatic path. A planted bug (percent / 10 instead of / 100) broke the pipeline. Duo fixed it by itself and left the test untouched. The governance agent labelled the MR, found no policy concerns, ran a verification pipeline, and it passed.
MR !2, the human-in-the-loop path. A fix targeting main was held as a Draft with [skip ci] until I clicked Approve in Slack. Only then did it run and pass.

How I built it

GitLab Duo Agent Platform: enabled the Fix CI/CD Pipeline foundational flow, created a pipeline-event trigger (on Failed), and enabled the Code Review flow.
Governance agent: Python + FastAPI. It receives GitLab pipeline and merge request webhooks, and the guardrails live in policy.py.
Human-in-the-loop: a Slack app with Block Kit Approve/Reject buttons, request-signature verification, and an optional approver allow-list.
Quality: 47 automated tests with fake GitLab, LLM and Slack clients, plus an MIT license.
Monitoring-ready: structured JSON event logs, plus scripts for Cloud Run deployment and Cloud Logging metrics and alerts.
Standalone mode: the agent can also diagnose failures itself with an LLM (Claude, Gemini, or a free mock), which is how MR !1 and MR !2 were produced.

Challenges I ran into

Duo didn't run its own tests. In MR !4, Duo wrote: "I couldn't run python -m pytest -q... I haven't run the real job." That confirmed the core idea: an AI fix needs independent verification, so the governance agent always runs a real pipeline.
Recognising Duo's work. Duo flows act with the triggering user's identity, so MRs show my name as the author. I detect Duo fixes by their duo/ branch prefix and [FixPipeline] title instead.
Setup puzzles. The flow was blocked by a group-level "foundational flows" setting, and Duo showed "No credits remaining" until I set the default Duo group.
Secrets hygiene. Tokens appeared in my own screenshots during development. I revoked and rotated them, and kept .env out of the repository with .gitignore.

Accomplishments that I'm proud of

A complete, working loop on real GitLab infrastructure: failure, Duo fix, governance, verification, then human approval.
Guardrails that stop the classic "make CI green" cheats.
Built entirely on free tiers.

What I learned

Autonomy is easy to add, and trust is the hard part. The most valuable piece wasn't the fix itself: it was verification, clear rules, and a visible audit trail.