-
-
The idea (illustration): every piece of release evidence must match one commit and one image digest, or the gate blocks.
-
Real record: our first release, v0.1.0, was blocked by our own gate. 109 HIGH, CRITICAL or UNKNOWN findings (job 17265824678).
-
The GitLab Duo agent's escalation issue #2: causes were outside its repair scope, it didn't patch. Explained why, gave the owner exact fix
-
After the owner's fix (MR !1): same gate, same policy, zero findings. Decision "passed" for commit caeadba and its image digest.
-
MR !3: the agent's write tools are pinned to one repair branch, with no shell. A test enforces this on every pipeline.
-
MR !7, written by the GitLab Duo agent: a read-only /policy in app/main.py, repair attempt 1 of 2. The gate passed.
-
Main pipeline 2987357926: gate, Cloud Run staging deploy, live smoke test, release report, then the owner's manual promote.
-
Production, live: the service publishes the gate policy its own release passed.
-
All nine lifecycle stages, each on a real record in the public project.
Inspiration
Release approvals are often built on evidence that no longer describes the thing being released. Yesterday's green test run, a scan of a different tag or an image rebuilt after review all look fine on a dashboard. We wanted a release step that cannot be satisfied by stale or mismatched evidence, and an agent that helps fix a blocked release without being able to quietly widen its own permissions.
What it does
- Deterministic gate.
evidence_validationcollects the test, package and scan results for one pipeline and checks that they all refer to the same commit SHA and the same container image digest. It blocks on any HIGH, CRITICAL or UNKNOWN finding. Its decision (passed/blockedwith a reason) and a human-readable report are stored as job artifacts. The same gate runs again before staging, before the release report and before production. - Release engineer agent (GitLab Duo flow). Mentioned on a release issue, it reads the real failing jobs, logs and scan artifacts. If the cause is application code, it repairs it with a minimal MR on a pinned branch. If the cause lies outside that scope (base image, dependency pins, CI or gate policy), it does not patch: it posts an escalation with the evidence and an exact fix.
- Owner stays in charge. The agent cannot merge, approve or promote. Its write tools are pinned in the flow definition to one repair branch, and the test suite fails if anyone adds a shell or an unpinned write tool. Production is a manual job that the gate re-checks: approver ID, protected branch head, staging revision and digest.
- Ships and watches. Main deploys the gated image to Cloud Run staging with keyless Workload Identity Federation, smoke-tests it live, and promotes the same digest on the owner's click. A daily scheduled monitor checks what production serves and re-scans the running image.
What actually happened (all records are public in the project)
- The gate blocked our first release. v0.1.0 (issue #2) was blocked by our own gate: pipeline 2960395835 found 109 blocking findings in the image.
- The agent escalated instead of patching. Duo session 9652104 read the job records and traced 106 findings to the Debian base image and 3 to a starlette pin. Both causes were outside its scope, so it escalated instead of patching. Elapsed time: 1 min 45 s.
- The recommended fix passed the same gate. MR !1 applied it: a Chainguard Python runtime pinned by digest, plus a starlette/fastapi bump. The owner reviewed and merged it. Same gate, same policy: zero findings, decision
passedfor that exact SHA and digest (pipeline 2960440396). - We pinned the agent's write access. The agent's service account could push to
main. MR !3 removed the shell and pinned every write tool toduo/release-repair, enforced by a test. - Eight days later, the gate caught new CVEs. On Oct 25 the same gate blocked a docs-only MR (!4): Trivy found 4 new HIGH CVEs, published after Oct 18, in the Chainguard base image (
ca-certificates-bundle,brotli). There was no exception and no policy change. MR !5 re-pinned the images to their patched digests, and the gate passed again (pipeline 2987286328). - The agent built a release itself. Issue #4 asked for v0.2.0: the service should publish the policy the gate enforces.
- The spec was written as tests (MR !6) and merged red on purpose (main pipeline 2987344287).
- A mention started Duo session 10025702. The agent read the failing job and judged the cause in scope (
app/main.py). - It opened MR !7 from its pinned branch: one file, repair attempt 1 of 2, about 1 min 40 s from mention to MR. The gate passed (pipeline 2987349589).
- Shipped to Cloud Run. The owner reviewed and merged. Main pipeline 2987357926 ran test → package → scan → gate → deploy_staging → smoke → release_report. The owner then clicked promote. The gate re-checked approver, branch head, staging revision and digest, and production now serves revision
nightjar-gate-00002-wl7, the same image that passed staging. - Configured and monitored.
infra/gcp-setup.shis the script that provisioned the cloud side: keyless trust limited to this project's protectedmain, production also needing theproductionenvironment, and three least-privilege service accounts. A daily scheduled monitor probes production and re-scans its running digest (first run: pipeline 2987406848, clean).
How we built it
Python / FastAPI service (Chainguard Python runtime); a standard-library gate and report generator. GitLab CI: test → package (Docker-in-Docker, digest captured, pushed to Artifact Registry) → scan (Trivy 0.66) → evidence_validation → deploy_staging → smoke → release_report → manual promote, plus scheduled monitor jobs. Google Cloud: Cloud Run (staging + production, min 0 / max 2 instances), Artifact Registry, Workload Identity Federation from GitLab's OIDC token, so there are no service-account keys anywhere. Every image and dependency is pinned by digest or hash. The Duo flow is defined in flows/release-evidence.yaml and published through the AI Catalog with a mention trigger. 115 tests cover the gate, the report, the CI structure, the monitor and the flow's tool pinning.
Lifecycle stages covered
Plan (release issues with acceptance criteria) · Create (MRs !1, !3, !5, and !7 written by the agent) · Verify (tests, junit report, live smoke test) · Package (image built once, pinned by digest) · Secure (Trivy with a blocking policy, re-run on the live image daily) · Release (evidence-bound gate, staging, owner-promoted production) · Configure (infra/gcp-setup.sh: WIF, service accounts, Cloud Run) · Monitor (daily probe + re-scan of production) · Govern (pinned agent tools, owner-only merge and promotion, test-enforced policy).
Challenges
- No per-tool approval. Custom catalog flows rejected per-tool approval settings, so supervision comes from pinned tool parameters plus owner merges.
- Limited project role. Our role cannot protect the production environment or attach variables to a schedule. So production is guarded by the manual job, a gate-checked approver allowlist and a WIF condition on the
productionenvironment claim, and monitor runs are keyed on scheduled pipelines. - Clean base images are hard to keep. Getting a base image with zero HIGH findings took several candidates, and a week later new CVEs arrived anyway. That is exactly why the gate re-checks every time.
What we learned
An agent that escalates with evidence is as valuable as one that patches. On day one the most useful thing it did was refuse to touch files it wasn't allowed to touch and explain exactly what the owner should change. A week later, given a spec as tests, it implemented the feature cleanly inside the same boundaries.
What's next
Automated rollback when the monitor fails, signed provenance attestations for each gate decision, and a separate policy project, so the gate code itself sits outside any branch an agent can touch.
AI use
- Development: an AI strategist (Claude) for concept, rule analysis, review and later code (MRs !1, !3, !5, the v0.2.0 spec, the cloud setup and the monitor), and an AI coding agent (Codex) for the initial implementation, tests and documentation.
- Runtime: the GitLab Duo Agent Platform powers the runtime agent: the escalation on #2 and the v0.2.0 implementation on #4.
- The entrant: chose the project and its name, reviewed and merged every change, linked billing for the cloud project, and promoted production. Details: AI-USE.md.
- Demo video: made locally with Remotion from real screenshots of this project; the narration is a synthetic voice (Kokoro-82M, Apache-2.0), and the video says so.