ReleaseGuard: Trustworthy AI-Fixed Releases

ReleaseGuard: when a pipeline fails, Duo fixes it automatically, a rule-based evaluator decides if the release is safe, an independent verifier checks the fix, and a human approves the merge.

Comment

Inspiration

AI agents can now make a failing pipeline go green. That raises a harder question: why should anyone trust that an AI-fixed release is actually safe? A green pipeline can be faked by deleting a test or loosening a rule. I wanted a workflow where the AI does the investigating and fixing, but never grades its own work.

What it does

ReleaseGuard closes the loop from failure to deployment:

  1. Evidence: the pipeline runs tests, SAST, a dependency audit, a Docker build and a Trivy image scan.
  2. Decision: a deterministic evaluator turns that evidence into a verdict (ready, caution or blocked) and saves a readiness.json that lists which rules passed, which failed, and what blocks the release.
  3. Agent: when the pipeline fails, a GitLab flow trigger starts Duo's Fix CI/CD Pipeline flow automatically. Duo diagnoses the failure and opens a draft merge request assigned to a human.
  4. Verification: the pipeline reruns on the fix, the verdict is recalculated, and a verify-fix job fails if the fix touched the release rules.
  5. Deployment: after a human merges, the image built for that commit is deployed to EC2 and checked with a health request. The verdict is a plain rule, not an AI opinion. With $B$ the blocking checks (tests, security, container), $A$ the advisory checks (dependencies) and $s_c$ the status of check $c$: $$ \text{verdict}= \begin{cases} \text{blocked} & \exists\, c\in B:\ s_c\neq\text{passed}\ \text{caution} & \exists\, c\in A:\ s_c\neq\text{passed}\ \text{ready} & \text{otherwise} \end{cases} $$

It runs at the Supervised level: detection, the fix and the verification are automatic, and a human makes one decision, merging the fix.

How I built it

  • A small Node.js (Express) app, containerised with Docker.
  • A GitLab CI/CD pipeline with SAST, npm audit, Trivy and a final release-readiness job that reads the real job results through the GitLab API and applies the rules in readiness.js.
  • The check jobs use allow_failure, so the evaluator is the single gate and the pipeline fails once, with reasons in the log.
  • A GitLab Duo flow trigger (pipeline failed, Fix CI/CD Pipeline flow) and a verify-fix CI job.
  • Terraform for the EC2 target, and a deploy job that runs only on main, only after release-readiness passes, and pulls the image tagged with that commit.

Challenges I ran into

  • Deciding who the gate is. Letting each scan job fail the pipeline made the verdict a mere report. Making the evaluator the one gate made it the real decision point.
  • Keeping the verifier honest. first verifier tried to detect every sneaky edit, and was brittle and confusing. I cut it down to two clear questions: did the fix change the rules, and is the verdict now ready?
  • Secrets in a public repo. A multi-line SSH key can't be a masked CI variable, so I base64-encoded it and stored it as a masked, hidden, protected variable that is only available on the protected main branch.
  • A restricted AWS account. A service control policy denied some EC2 calls, so I had to work out the difference between missing IAM permissions and an organisation-level deny.
  • Deploy-script details. Quoting, an SSH client missing from the image, and commands running on the runner instead of on the server all broke early versions of the deploy job.

Accomplishments that we're proud of

  • Duo starts on its own. A flow trigger on "pipeline failed" launches the Fix CI/CD Pipeline flow with no button press. It diagnosed the intentional failure and opened a draft merge request with a one-line fix.
  • The AI never grades its own work. The release decision comes from a rule-based evaluator, and a separate verify-fix job checks that the fix didn't change the release rules and that the verdict is ready.
  • Real evidence, not sample data. The evaluator reads actual job results from the GitLab API, and the deploy job only runs on main after release-readiness succeeds.
  • Supervised by design. Detection, fix and verification are automatic, and a human makes one decision: merging the fix. ## What we learned
  • A pipeline failure is a good trigger for an agent, but the agent needs an independent check, not self-reporting.
  • GitLab's flow triggers make Duo event-driven, so the human steps in at the decision, not at every step. ment that limit instead of overclaiming. ## What's next for ReleaseGuard: Trustworthy AI-Fixed Releases
    • Block on SAST and Trivy severity, not job status.
  • Replace SSH with GitLab-to-AWS OIDC and a locked-down network.
  • Add a manual approval gate before a production environment.

Built With

Share this project:

Updates

Submission history