lights-out

Agents propose, numbers decide. A governor for GitLab CI gives each agent-written change a level of autonomy, then lets bot players judge the canary and roll back on their own.

Comment

Inspiration

AI agents now open merge requests faster than anyone can review them. Teams fall back on one of two defaults: a human checks everything and becomes the bottleneck, or the agent ships everything and green tests become the only safety net. But tests can pass and still miss the change, and a canary nobody looks at doesn't protect anyone.

I wanted the question "how much can this change do alone?" answered by evidence, change by change, and never by the model.

What it does

lights-out governs a real, live game, OVERBOOKED!, a co-op game for people who run tech events that I built and people already play. It runs on GitLab:

  • Agents propose. GitLab Duo flows turn an issue into a spec, a declared perimeter (the only files the change may touch) and a merge request with tests. Other flows write the tests the mutation gate asks for, write the go-for-launch brief and triage incidents. When the planner was unsure whether a URL counted as a forbidden id, it stopped and asked a human instead of guessing.
  • Numbers decide. Seven gates print one line each, GO or NO-GO: no third-party marks, the game's own 100 checks with playtest difficulty bands, 84 spec tests, the perimeter, mutation testing of the changed lines only (tests must kill 80% of the mutants), GitLab SAST and secret detection against a triaged baseline, and the canary.
  • Autonomy is a policy. governor/policy.yml gives each change a level from the files it touches. Texts ship hands-off. The simulation is supervised: a human approves the promotion from a brief. Security-sensitive files and the governor itself are assisted. Big changes, a deploy freeze, a recent rollback or an author whose changes keep failing take autonomy away. Every threshold says where it comes from.
  • Players judge the release. For each merge the governor writes the release pipeline. A signed image goes to Cloud Run as a canary at 5%, then 25%. Bot players open the public URL and play whole days on the canary and on stable with the same seeds. If errors, completion or player feedback get worse, the pipeline rolls back on its own.
  • It watches production. Every hour the bots play production; a missed objective opens an incident for the triage flow.

What happened on GitLab

  • Hands-off (!2): a text change from an agent, all gates GO. The governor merged it, and the release pipeline went to 5%, 25% and 100% with no manual job.
  • A regression the gates could not see (!4): a "lighter page" build, declared as a seeded regression, with every gate green because they all run on the sources. A human approved it. At 5%, stable completed 100.0% of days with 0.0% errors; the canary completed 0.0% with 100.0% errors. NO-GO, and the pipeline rolled back on its own.
  • The governor governs itself (!3): GitLab's scanners skipped merge request pipelines, so the security gate had nothing to judge. The fix went through the same pipeline, as assisted, and a missing report is now a NO-GO.
  • Production: the bots played production 287 times in 12 days. One run missed the objective (1 of 30 sessions got an HTTP 500), opened an incident, and the ops-triage flow triaged it.
  • Mutation testing that fits in CI: a weekly bonus with no test of its own got NO-GO: mutation, 78% of 9 mutants killed (min 80%); with one test, GO. Mutating only the changed lines takes seconds; the whole scope is 9,334 mutants, about 9 hours.

How I built it

  • GitLab: merge request pipelines, SAST and secret detection, child pipelines written by the governor, environments, labels, schedules, issues as incidents, and Duo Agent Platform (four custom flows in the AI Catalog, the flows API, the Code Review flow with our instructions, AGENTS.md and skills).
  • Google Cloud: Cloud Run (tagged revisions, traffic splitting), Cloud Build, Artifact Registry, Cloud Logging, and Workload Identity Federation, so no key is stored anywhere. Images are signed keyless with Sigstore using GitLab's OIDC token and verified before any traffic.
  • Code: Node.js with no runtime dependencies, node:test, StrykerJS.

Stages covered

Plan (issues to specs and perimeters), Create (agent merge requests, Duo code review), Verify (seven gates), Secure (SAST, secret detection, triaged baseline), Package (Cloud Build, Artifact Registry, signed images), Release (generated child pipelines, canary, automatic rollback), Configure (policy as code, OIDC federation), Monitor (hourly bots, incidents, triage flow), Govern (autonomy levels, guards, labels and a governor note on every merge request).

Challenges

  • The whole-scope mutation run would have taken about 9 hours, so the gate mutates only the changed lines, and the policy says so.
  • Tuning the bar: at 0.60 a bonus with no test of its own passed, so the threshold went to 0.80, with that reason written next to it.
  • The security gate first counted 137 old findings. It now compares against a baseline where each accepted finding has a reason.
  • My first canary failed because the new revision wasn't routed yet. The deploy step now waits until the canary answers with the expected build.

What I learned

The gates that matter most are the ones that say NO. A rollback happens only if the canary looks at what players get, not at the sources. And an agent that stops to ask a precise question is worth more than one that guesses well.

What's next

Let the strengthen-tests flow run on GitLab when a change actually needs it (so far none has), configure the online mode, and give agents autonomy from their own track record over more changes.

Declared limits

The strengthen-tests flow is wired to the mutation gate but has not run on GitLab yet: the mutation NO-GO above was measured locally. Firebase is not configured, so online rooms run between tabs and stats and boards are off. The bots exercise the game's logic, not its rendering.

Built With

  • artifact-registry
  • cloud-logging
  • gitlab
  • gitlab-ci
  • gitlab-duo
  • google-cloud-build
  • google-cloud-run
  • node.js
  • sigstore
  • stryker
  • workload-identity-federation
Share this project:

Updates

Submission history