Phoenix

A release passes every test, then breaks for real users. Next month, the same bug ships again.

Phoenix closes that loop on GitLab. It catches the bad release at 10% exposure, switches it off 28 seconds later with no human, files the incident, opens the fix, and turns the lesson into a review rule that catches the same bug in the next merge request.

What it does

One change travels through all nine GitLab stages, plus a Learn step:

1. Plan. The phx-spec flow adds acceptance criteria and a risk rating to the work item.
2. Create. The phx-build flow writes the branch, code, tests and merge request, with the new behaviour behind a feature flag.
3. Verify and Secure. CI runs lint, tests, SAST, secret detection and a policy check. GitLab's Code Review flow reviews against the project's own rules.
4. Merge. A human merges. That is the Supervised gate: Phoenix never merges anything itself.
5. Package and Release. CI builds the image, makes an SBOM, tags a version and creates a GitLab Release.
6. Configure and Monitor. CI rolls the flag out at 10, 50 and 100 percent. After each step, a canary sends 60 seconds of traffic and fails closed.
7. Rollback. If the canary fails, a CI job switches the flag off and reads it back. No human, no LLM.
8. Incident. The phx-incident flow reads the failed job, files an incident and opens a fix merge request with a regression test.
9. Govern. Every release gets a checksummed evidence pack: tests, scans, SBOM, policy result, rollout, rollback and links to the AI sessions.
10. Learn. The phx-retro flow turns the closed incident into a post-mortem, regression tests and a new review rule. On the next merge request, GitLab's Code Review flow cited that rule, named the TypeError the change would raise, and requested changes.

Flows propose, CI executes

AI flows read messy text and write code, comments and issues. Nothing that merges, releases, runs a pipeline or flips a flag is in their hands. Those steps are plain, unit-tested CI scripts. A validator enforces this on every pipeline:

• A component that reads untrusted text (issues, logs) never holds a write tool, and every prompt labels outside text as untrusted.
• No flow may hold a tool that runs commands or calls the generic API.
• A flow's merge request may only touch app/, tests/ and docs/ (plus the review rules, for the Retro flow), within size limits, and code needs tests.
• Feature flags fail closed. Auto-merge is off.

Built on GitLab Duo Agent Platform

Results, live on GitLab.com

Challenges

• The review rules were hidden from the flows. GitLab's context exclusion hides .gitlab/duo/, so the first Retro run stopped safely without changing anything. We added a readable mirror of the rules, kept identical by a CI check, and the next run wrote the rule.
• Flows are not deterministic. One Build run forgot to register its feature flag, so we closed that merge request and ran Build again.

What's next

• A real deployment target. Today the canary runs the released image as a CI service against the live GitLab feature flags, and it can already point at a real URL.
• Locking down main, so the policy check always runs from a protected copy.

See it run

• Bad release caught and rolled back (run B pipeline)
• Incident flow session and the incident it filed
• Fix merge request
• Retro merge request with the new rule
• Review citing the rule on the next merge request
• Build session, its merge request, release pipeline and evidence pack (run A)

Run B's fault was planted for the demo, and the second merge request in run C was written to test the new rule.

Built With

• ci/cd
• cyclonedx
• docker
• fastapi
• feature-flags
• gitlab
• gitlab-duo
• mcp
• pytest
• python
• ruff
• sast
• sqlite
• unleash