Flagfall

Every feature flag is born with an exit plan. GitLab Duo agents register it, roll it out, roll it back, remove its dead code with proof, and delete it from GitLab in the right order.

Comment

Inspiration

On 1 August 2012, Knight Capital deployed new trading code that reused an old feature flag. One of its eight servers never got the new code, so on that server the flag woke up code that had sat unused since 2003. Forty-five minutes later the firm had lost more than $460 million.

Feature flags are how teams ship safely. New code goes to production switched off, then reaches 10%, 50%, 100% of users. The last step, deleting the flag, is the one nobody does. I measured it on GitLab's own codebase: for each flag, I took the merge date of the merge request that introduced it and compared it with the lifespans in GitLab's feature flag guidelines. 200 of 634 flags are past that lifespan, even with two months of grace. About one in three.

Nobody is careless here. Once a flag ships, nobody owns it anymore. GitLab Feature Flags has no owner field, no deadline and no idea that a flag is stale. Cleaning up is post-code work on a schedule known in advance, which is exactly the kind of work I wanted agents to own.

What it does

Flagfall gives every flag a birth certificate and an exit plan. Three GitLab Duo flows carry the plan out, from the merge request that adds the flag to the moment the flag disappears from GitLab.

  • Birth. A merge request adds isEnabled("some-flag"). The Registrar flow reads the diff, writes the flag's certificate (flags/<name>.yml: owner, type, success metric, rollout steps, deadline, final value) and opens the rollout issue. The pipeline fails any flag in the code that has no certificate.
  • Rollout. Every hour, a scheduled pipeline sends real traffic to production and splits errors and latency by the variant each visitor got. Code turns the numbers into a verdict: advance, hold or roll back. On "advance", the Rollout flow opens a one-line merge request to the next step of the plan.
  • Rollback. On "roll back", the pipeline switches the flag to 0% at once, without a redeploy. Then the Rollout flow reads the recent commits, finds the root cause and opens a bug issue with the fix.
  • Retirement. When a flag has been at its final value for 3 days, or gets close to its deadline, the Retirer flow removes the flag and the dead branch of the code, runs the type check and the tests, and opens the merge request.
  • Proof. The pipeline replays the same shopping scenarios on the old code (flag forced to its final value) and on the new code (flag removed). If both outputs have the same SHA-256, users can't tell the difference.
  • Deletion. Flagfall deletes the flag from GitLab Feature Flags only after the merge is deployed and production reports that its code no longer reads the flag. Done the other way round, the old code would ask for a flag that no longer exists, get OFF, and switch the feature off for everyone.

Agents never change a flag directly. They edit certificates through merge requests, and one CI job applies them to GitLab Feature Flags. So every percentage change has a merge request, a pipeline and a line in the certificate's history.

It ran for real

Everything in the video happened on 13 and 14 October 2026, in the public project, against a small demo shop running in production on Vercel. Here is one flag's whole life, recommendations-v2:

  1. I opened merge request !4, which adds "goes well with" recommendations behind the flag. The Registrar committed the certificate and opened issue #6.
  2. The Rollout flow moved the flag to 10% (!5). At 10%, the canary measured 2 errors in 70 requests with the flag ON and 0 in 570 with it OFF: 2.9% against a 1% limit. The flag was back at 0% two seconds after the measurement.
  3. The Rollout flow found my bug, a line that assumed the cart is never empty, and opened issue #7 with the fix. I handed the issue to GitLab's Developer agent, which wrote the patch and three tests (!6). Bug fixes are not on Flagfall's hands-off list, so I reviewed and merged that one myself.
  4. The rollout started again from 0%: 10% with 0 errors in 54 requests, 50% with 0 in 322, then 100% (!7, !8, !9). Flagfall merged each step itself once every guardrail passed.

Two more flags went all the way to deletion. legacy-coupon-engine plays the zombie: switched off and seeded as three months past its deadline. The Retirer removed it in !1, the proof replayed 210 scenarios with an identical SHA-256, and the next release deleted the flag from GitLab once production build 9430f723 stopped reading it. free-shipping-35 was an experiment that won; it was retired in !2.

More than 90 pipelines have run so far, most of them green, including the hourly scheduled runs.

How I built it

  • GitLab Duo Agent Platform. Three custom flows (.gitlab/duo/flows/), published in the AI Catalog and enabled in the project, each with its own service account. A "merge request created" trigger starts the Registrar. The scheduled pipeline starts Rollout and Retirer by mentioning them on the rollout issue.
  • Deterministic code for everything that must not depend on a model: the canary, the verdicts, the guardrails, the equivalence proof and the deletion order (flagfall/src, TypeScript, 22 tests).
  • GitLab CI/CD: verify (type check, the shop's tests in both states of every flag, certificate and expiry check), SAST and secret detection, the equivalence proof on retirement merge requests, release (build stored in the package registry, deploy to Vercel recorded as a GitLab environment, build SHA confirmed in production, flags applied), the hourly operate run, and a registry page of every flag.
  • GitLab Feature Flags through its Unleash API. The shop (Next.js) reads flags with the Unleash client, and its /api/flagfall endpoint reports the live build and the flags its code reads.

Guardrails. Flagfall merges an agent's merge request with nobody in the loop only if all of these hold: the branch is an agent branch; the pipeline for the latest commit passed; a rollout merge request changes one certificate, only its percentage, stage and history, and moves exactly to the next planned step; a retirement merge request touches only the shop's source and its own certificate, never the scenarios, and has a successful equivalence proof. Anything else goes to a human, with a comment that says why.

Security. The Registrar is two agents. A reader extracts JSON facts from the merge request, which is untrusted input. A writer acts only on those facts, so instructions hidden in a diff never reach the agent that can commit. The Retirer runs commands in the checkout with network access limited to the npm registry.

GitLab stages covered: all nine

  • Plan: the Registrar opens the rollout issue with the plan; the scheduled job plans retirements
  • Create: the Retirer writes the dead-code removal; the Registrar writes the certificate
  • Verify: type check, tests in both states of every flag, certificate and expiry check, equivalence proof
  • Secure: SAST, secret detection, reader/writer split against prompt injection, network policy for agents
  • Package: the shop build is versioned in the GitLab package registry
  • Release: production deploy recorded as a GitLab environment, confirmed by the build SHA production reports
  • Configure: flags/*.yml applied to GitLab Feature Flags (GitOps)
  • Monitor: canary in production per variant, automatic rollback
  • Govern: guarded hands-off merges, lifespans and deadlines, audit trail in every certificate, flag registry page

Challenges I ran into

The agents' token can't read the Feature Flags API. That limit became the design: agents write YAML through merge requests and CI applies it. It's also a better audit trail.

GitLab lets only humans start a flow, so the pipeline posts its mentions with my token, and the author of a comment proves nothing. The Rollout agent therefore doesn't trust the comment. It checks the verdict against the CI job log, which nobody can forge from a comment box.

Agents have no clock. The first Registrar run stopped and asked me for today's date. Every date now comes from GitLab: the merge request's creation time, the canary note's timestamp, or date -u in the Retirer's shell.

GitLab computes mergeability lazily: right after a push, a merge request reads "unchecked". The guard waits for a real "mergeable", merges one merge request per run, and rebases the others for the next run.

Accomplishments that I'm proud of

A real rollback in production: root-caused by one agent, fixed by another, rolled out again to 100%, and every step is a merge request, an issue or a pipeline you can open.

Two flags removed from the code and from GitLab, each with a proof that users saw no difference.

Deleting a flag too early is a real outage, and Flagfall can't make that mistake: it asks production first.

What I learned

Agents are good at the judgement work: reading a diff, writing a dead-code removal, explaining a crash from a stack of commits. I don't want them to be the safety mechanism. Every safety decision in Flagfall is code with tests, and the agents work inside it.

Trust between agents has to be checked too. The Rollout agent re-reads the CI log before it acts on a verdict, even though the verdict came from my own pipeline.

What's next

Other flag strategies (user lists, A/B tests with a real statistical test), a flag finder for more languages than TypeScript, and deadline extensions requested in the issue ("+30 days, because…"), which the certificates already support with a mandatory reason.

A note on the demo

The demo shop sells nothing and its products are made up. In the certificates, the history before October 2026 is seeded demo history, such as the zombie flag's old deadline. Everything from 13 October 2026 on is real runs, linked from the README.

Built With

Share this project:

Updates

Submission history