Release Gate blocks the next agent MR until a human approves (local replay)
Autonomy ledger after drill #1 (local replay): one Defender miss drops verify to Supervised
Drill #1 (local replay): five planted faults, F1 to F4 caught

Inspiration

Teams decide how much autonomy to give coding agents by gut feel. Nothing measures whether the agents catch what matters, and nothing takes autonomy back when they do not. We wanted "can we turn on auto-merge for the agents?" answered per stage, from evidence, for platform leads at regulated and fintech companies.

What it does

Fire Drill is designed as a GitLab Duo Agent Platform system. A Saboteur plants deterministic faults in a small payments-ledger API. A Defender review agent and CI jobs must catch them. Each catch or miss is scored per (fault, catcher) and written to an autonomy ledger, policy/autonomy.yml, on main. A Release Gate reads it: at hands_off, with required jobs green and a Defender APPROVE for the head commit, an agent-authored merge request auto-merges; at supervised it needs one human approval. A miss demotes a stage one level (two for high severity); two clean drills promote it.

Replayed locally, the story runs end to end: a hands-off merge and promote with no approval event; drill #1 catches F1 to F4, the Defender stand-in misses F6 (a silent overdraft) and the unit job catches it, so verify drops to supervised; the next agent MR is blocked citing that demotion; after a hand-written repair, drills #2 and #3 block F6 and verify returns to hands_off. The harness replaces GitLab and Cloud Run with git branches and a local uvicorn process, and its Defender is a rule engine, not a model.

How we built it

FastAPI and SQLite ledger with invariants I1 to I4; 93 tests, including a Hypothesis property test.
Seven fault scripts, five active across verify, secure, package and configure.
Python modules for the engine, scoring, gate and dispatch. Merges and ledger writes are CI jobs using a project token; no agent identity merges anything. Gate and ledger are read from main, so an MR cannot edit them to approve itself.
.gitlab-ci.yml, six Duo flows, Cloud Run deploy config and a Workload Identity Federation script, written from documentation.
Checks: 468 tests pass; the local story passes 36 of 36 checks in 607 s.
On GitLab, main pipelines are green (first 2986687701, latest 2986823616), job logs public: unit tests and drill self-tests (468 in GitLab's test report, 3 semgrep tests skipped), the invariants replay, GitLab SAST, Secret Detection and Dependency Scanning, our custom semgrep rule, the three scanner gates, config drift, the buildah image build and the Pages scoreboard.

Status: the CI pipeline runs on GitLab. The drill has not run there, the Duo flows are registered but GitLab refused to start them for our project token, and nothing is deployed to Cloud Run. The story above is the local replay.

Challenges we ran into

The PRD's Flows API path and label or comment triggers are not in GitLab's docs, and bot-posted mentions do not fire triggers. A scheduled dispatch job calls the Flows API instead; whether a project-token user may is unconfirmed.
Scanner jobs exit 0 on findings, which the scorer would read as misses. Three gate jobs now fail on findings or a missing report.
GitLab's secrets analyzer flagged our own docs for containing the example key, so they write it truncated.
F1 is high severity and would demote verify two levels, so the engineered demotion uses F6, whose miss depends on a deliberate checklist gap.
GitLab.com's account identity check blocked shared runners, so the pipelines ran on a temporary project runner on our machine, removed afterwards.
The first live runs found real bugs the local harness could not: semgrep and fastapi pin conflicting opentelemetry versions, the slim Python image has no git for the harness tests (installing it with apt-get hung one job), buildah needs seccomp:unconfined on a self-hosted runner, and pip inside the buildah build needed retries for PyPI timeouts.
GitLab rejects flow prompts that contain an HTML comment, so two flows spell out their marker in words.
Starting a flow through the Flows API with the project token returned "forbidden to access duo workflow".
Cloud Run needs a billing account, and we kept Google Cloud billing off, so nothing is deployed.

Accomplishments that we're proud of

The whole PRD story replays locally, 36 of 36 checks.
Credit goes per (fault, catcher): a fault CI caught but review missed still demotes review.
Gate hardening: verdicts bind to the head commit, governed paths need a human, errors fail closed.
A per-claim honesty table: verified, stand-in, not run.

What we learned

A receipt is not a result: a skipped scanner is not a pass, and a merge call is not a merge.
The local replay shows the control loop works; it measures nothing about an LLM reviewer.
Read the platform docs before fixing the design.

What's next for Fire Drill

Add the project token as a CI variable and run the drill on GitLab, so the ledger change lands on main from a real pipeline.
Turn on the group's experiment features and start the flows as a user that passes Duo's access check, then attempt a zero-touch run.
Run the real Defender against F6 and see whether it misses.
Deploy to Cloud Run through Workload Identity Federation once a billing account is available.