AegisFlow

• 

  The nine stages AegisFlow runs for every incident, from detecting the failure to a person approving the release.

• 

  AegisFlow's dashboard after a real recovery: all nine stages are evidenced, and a repair merge request opened 13 minutes after the alert.

• 

  The recovery policy sets the limits: two attempts, 180 changed lines, and the checks that must pass.

• 

  A real Cloud Monitoring alert for the faulted version, and the GitLab incident AegisFlow opened with the evidence attached.

• 

  GitLab Duo investigating the incident, reading the evidence through AegisFlow's own MCP server.

• 

  Duo's repair, with a new test that fails on the broken code and passes on the fix.

• 

  The release pipeline: package, configure, test as a canary, then promote after a person approved.

• 

  A software bill of materials and an image scan, produced for the exact container that shipped.

• 

  The repair had to pass all 11 required checks, including the security scans, before it could move forward.

• 

  GitLab Duo also reviewed AegisFlow's own code, caught real issues, and approved once they were fixed.

• 

  The canary test: 120 requests over 60 seconds with no errors, confirmed independently by Cloud Monitoring.

• 

  Delivery on Google Cloud: keyless sign-in from GitLab, the Cloud Build that built the image, and that exact image in Artifact Registry.

• 

  The repaired version now serves all production traffic, and the request that used to fail returns the right 400 response.

• 

  What the agent did, and the release gate: promotion waited for a person to approve the exact version.

• 

  Every step is recorded in a tamper-evident timeline, so what happened and who approved it can be checked later.

Inspiration

When an application breaks, finding the bug is only the beginning. Someone still has to fix it, prove the fix is safe, and get it back online, usually under pressure and often at night. AI agents can now write code, but most teams won't let one near production, because "the agent said it works" is not evidence. We wanted to see how far an agent could take a real incident if every step had to be proven, bounded and auditable.

What it does

AegisFlow turns a production alert into a traceable path to recovery, using GitLab Duo and Google Cloud:

1. Detect. Cloud Monitoring catches a failure on a Cloud Run revision. An authenticated Pub/Sub push reaches the AegisFlow controller, which deduplicates repeat alerts into the same incident.
2. Plan. The controller opens a GitLab incident issue with the Cloud Logging evidence and a suspected cause.
3. Govern. A version-controlled policy bounds the agent: at most 2 attempts, 5 files and 180 changed lines, frozen dependencies, and a fixed list of required checks.
4. Create. The controller starts our custom flow in the GitLab AI Catalog through the Flows API. A diagnose agent reads the incident through AegisFlow's own MCP server; a repair agent writes the fix plus a new regression test and opens a linked merge request.
5. Verify and secure. Race tests, HTTP integration probes, controller tests, SAST, secret detection, dependency scans and an explicit fail-closed security gate must all pass on the exact commit. The controller also checks the Duo session trace itself, not the agent's description of it.
6. Package, configure, release. A protected pipeline builds an immutable image with Cloud Build, records a CycloneDX SBOM, scans it with Trivy, applies bounded Terraform, and sends 10% of traffic to a canary that must serve 120 requests over 60 seconds, confirmed independently by Cloud Monitoring.
7. Approve and promote. In supervised mode a person approves that exact commit before production moves to 100%. Then the controller closes the incident.

Every step is written to a hash-chained ledger and mirrored as notes on the incident, and a public dashboard shows what happened, what the agent changed, which checks passed and where a person stepped in.

The live recovery

We ran this end to end on real infrastructure:

• A disclosed fault, deployed to a tagged revision with zero production traffic, made /base64 panic with HTTP 500. Cloud Monitoring raised a real alert and the controller opened incident #3.
• Duo session 10053574 read the evidence over MCP and opened MR !1: a general fix returning 400 for bad input, plus a regression test that fails before the fix and passes after. That was 13 minutes after the alert. All 11 required checks passed.
• The first release stopped at Package: the image gate denied three Debian 12 advisories in the base image. Instead of weakening the gate, we built release remediation: the controller accepts a follow-up commit only if the shipped application is byte-identical to the verified repair and only the base image FROM line changed. MR !3 moved to Debian 13 distroless.
• Release pipeline 2992519572 then passed all 16 jobs. The canary served 120 requests in 60 seconds with 0 errors and a 209 ms p95. We approved the exact commit, production moved to 100%, and the request that used to crash now gets a clean 400.

How we built it

• Application: go-httpbin by Will McCutchen (MIT), preserved at baseline 13bbded, deployed on Cloud Run with a small adapter for structured failure logs.
• Controller: Python and Flask on Cloud Run, with a Firestore ledger, Cloud Scheduler reconciliation and a read-only public dashboard.
• Agents: a custom GitLab Duo flow (diagnose and repair components) registered in the AI Catalog, started via the Flows API, plus an MCP server built on the official Python SDK.
• Delivery: GitLab CI/CD with keyless Workload Identity Federation to Google Cloud (no stored keys), Cloud Build, Artifact Registry, Syft, Trivy, Terraform and Cloud Run traffic control.
• Review: GitLab Duo Code Review reviewed AegisFlow's own merge requests and caught real bugs before approving, including a Dockerfile parser-directive bypass in our remediation check and a private-detail leak in the public API.

What we added (Path B)

Everything except the upstream application is new: the controller and ledger, the Duo flow and prompts, the MCP server, the CI/CD and security gates, release remediation, the Terraform for Google Cloud, the dashboard, and 146 controller tests. Upstream licensing is untouched; Duo's repair and our one compatibility fix are attributed separately in the repository.

Challenges

• GitLab's composite identity attributes agent MRs to the human who started the flow, which our first attribution check rejected. We now verify the session trace and pipeline identity instead.
• A trailing newline in a stored token broke the first Plan; operators can now reconcile a stalled stage without resetting attempt budgets.
• The base-image advisories blocked the release after merge, which led to release remediation and a new pre-merge base-image scan.
• Upstream timing tests are flaky under -race on small runners, and we ran out of shared CI minutes mid-project and moved CI to a self-hosted runner.

What we learned

An agent's narrative is not evidence. The interesting engineering is in the gates: exact-commit checks, independent metrics, bounded policies, and a ledger that records failures as faithfully as successes. Keeping humans in charge of how much control to hand over turned out to matter more than the agent's raw ability.

What's next

Exercising hands-off mode and automatic rollback on live incidents (both implemented and tested, not yet run live), registering the release-remediation flow so Duo also fixes base-image blockers, and broadening fault types beyond request handling.

Built With

• artifact-registry
• cloud-build
• cloud-logging
• cloud-monitoring
• cloud-run
• duo-agent-platform
• firestore
• flask
• gitlab
• gitlab-ci
• gitlab-duo
• google-cloud
• google-cloud-run
• pub-sub
• secret-manager
• semgrep
• syft
• trivy
• workload-identity-federation