ReleasePilot AI

Inspiration

When a CI pipeline fails, an engineer has to dig through logs, find the cause, write a fix, and then decide whether the fix is actually safe. AI agents can do the first two steps, but most "AI fixes your CI" demos then let the agent declare its own success. We wanted to build the missing piece: let the agent propose, and let plain, deterministic code decide.

What it does

ReleasePilot AI recovers from failed GitLab CI/CD pipelines.

1. A GitLab Duo Agent Platform custom flow (Investigator → Recovery Engineer → Release Assessor) reads the failure and commits a minimal fix, plus a regression test, to its own branch.
2. CI re-runs on that exact commit.
3. A deterministic policy engine collects the real CI results and decides: ready for human approval or blocked. It fails closed on anything failed, missing, stale, from a different commit, or beyond one repair attempt.
4. A patch-safety check reads the actual diff and blocks fixes that edit CI or policy files, delete tests, lose assertions, or add suppressions like allow_failure or --ignore-vuln.
5. A dashboard shows the failure, the agent's diagnosis, the diff, every check and an audit timeline.

The agent has no merge, deploy, shell or approval tools. Production approval stays with a person in GitLab.

What we verified on real GitLab runs

• Failing unit test: the agent fixed a sign error and added a regression test.
• Vulnerable dependency: pip-audit reported 5 vulnerabilities in jinja2 2.11.2. The agent chose 3.1.6, the one version that clears every finding (lower versions leave some open).
• Prompt injection: a failing test's output told the agent to delete the tests, set allow_failure: true and push to main. The agent ignored it, quoted it as a suspected prompt injection, and made the minimal real fix.
• A cheating fix (simulated by hand): deleting the failing tests turns CI green with the bug still present. The patch-safety check blocks it.
• A wrong fix (simulated by hand): blocked because unit_tests fails.
• Unreadable evidence: when the agent could not read the scan output, it changed nothing and committed a report saying exactly which step failed.

How we built it

Python 3.12, FastAPI, pytest, a Next.js + TypeScript dashboard, GitLab CI/CD, and a Duo Agent Platform flow generated from versioned prompts. Evidence is normalised from GitLab's API into JSON records, with log cleanup and secret redaction before anything is stored.

Challenges

• The agent's API tool could not read job logs or artifact files (it returned empty bodies). We routed evidence through GitLab's JSON pipeline test report instead, converting pip-audit output into JUnit with a small script.
• Silent failures. Early runs ended without a result. We found a tool the flow was not allowed to use, and we made the agent always end with a written explanation.
• A rule that looked right and was wrong. Our first dependency prompt said "use the lowest fixed version", which would clear only one of five findings. We caught it before spending an agent run.
• "Never remove tests" was only a prompt. We realised a fix that deletes the failing tests would pass CI, so we built the diff-based patch_safety check and proved it on a real cheating fix.

What we learned

Prompts are requests, not controls. The useful design is an agent that is allowed to be wrong, wrapped in checks that cannot be talked out of their answer.

Honest limitations

It runs on a small sample service, with two failure types demonstrated (unit tests and dependency vulnerabilities). Docker-build, configuration and deployment failures are planned, not built. There is no staging deployment or in-app approval, by design. The dashboard runs locally and has no authentication. We make no accuracy or time-saved claims; the one timing we measured (4 minutes from failure to proposed fix) is a single run.

What's next

Docker-build failures, a real GitLab protected-environment gate driven by the policy, and automatic posting of the verdict on the merge request.

Built With

• fastapi
• gitlab-ci/cd
• gitlab-duo-agent-platform
• gitlab-secret-detection
• junit
• next.js
• pip-audit
• pytest
• python
• react
• ruff
• typescript