Pre-mortem-agents

Inspiration

Agents can write, review and deploy code, yet teams still don't let them ship alone. The reason is trust: nobody asks "what will this break?" before a change goes live. The tests and alerts that would have caught an incident usually get written in the post-mortem, after the outage.

Reliability engineers have an old answer to this: the pre-mortem. Assume the change has already shipped and caused an incident, then work backwards to why. It works, but it takes time, so nobody does it for every merge request. Agents can.

AI risk review and canary analysis already exist on their own. What's new here is the coupling: every predicted incident becomes a machine check. It runs against a live deployment of the MR before merge, against the canary before release, and on every release after that.

What it does

For every merge request, with no human in the loop:

1. Predict. An oracle agent (GitLab Duo custom flow) reads the diff and the code it touches. It writes the post-mortems of the 1–4 incidents the change is most likely to cause. Each one carries a check: p95 latency, 5xx rate, sensitive data in a response, or a status code.
2. Defend. A builder agent commits a regression test and a Cloud Monitoring alert policy for each prediction. When the defect is plain in the diff, it also commits a minimal fix. It then posts the pre-mortem on the MR.
3. Prove. The MR is deployed as its own Cloud Run revision (0% traffic, its own tag URL), and every prediction is checked against it. A materialized prediction blocks the merge.
4. Release. On main, every prediction ever made is replayed against a 0% canary. The canary then gets 10% of traffic, and its p95 and 5xx rate are read from Cloud Monitoring. Within limits, it is promoted (pinned), alert policies are applied, and a release is created with the report as evidence. Outside them, the release rolls back automatically.
5. Close the loop. A reporter agent posts "The incident that never happened" on the MR, or opens an incident issue whose post-mortem was written before the incident. It records the outcome on a scoreboard that the oracle reads before its next pre-mortem.

How we built it

• GitLab Duo Agent Platform.
  • Two custom flows with three agents (oracle, builder, reporter), triggered on MR creation and on pipeline events.
  • A tool option pins the reporter's commits to a scoreboard branch.
  • A custom MCP server gives Duo Chat the same data: predictions, live verification, scoreboard, Cloud Run state and canary telemetry.
• GitLab CI/CD across all nine stages:
  • plan, create, verify, package, secure (SAST, secret detection, pip-audit);
  • release (per-MR revisions, auto-merge, canary, promote or rollback);
  • configure (alert policies as code);
  • monitor (Cloud Monitoring canary watch);
  • govern (release evidence, and only commits from merged MRs may deploy).
• Google Cloud: Cloud Run (revision tags, traffic splitting), Cloud Build, Artifact Registry, Cloud Monitoring. Authentication uses Workload Identity Federation, so no service-account keys exist anywhere.

Challenges we ran into

• The rollback itself had a bug. promote used --to-latest, so a later MR preview became production. During !4, the rollback restored that slow preview, and production served it for about 30 minutes until traffic was restored by hand. We corrected the agent's incident report in #2. Promote now pins the exact canary revision.
• Latency measured from CI lied. The same healthy canary measured p95 876 ms from a runner and 132 ms from a laptop. Checks now use the app's own Server-Timing header.
• We couldn't protect main. Agents run as Developers, who can push to main in this workspace, and branch protection can't be changed. So production is protected instead: only commits that arrived through a merged MR can deploy.
• Least privilege kept biting. Cloud Build logs need project Viewer, so builds log to Cloud Logging instead. Google's 127-byte subject limit broke on a long branch name, so the WIF subject is now project id plus job id.

Accomplishments that we're proud of

The whole loop ran hands-off on real merge requests, and everything below is public in the repository:

Where it falls short: a teammate, working with his own AI assistant, found a stock bug: two order lines for the same product could oversell it. No agent predicted it, because none of our check types covers multi-step business logic. TODO: benchmark results on blind predictions, misses and false blocks.

What we learned

Agents were best where the incident was concrete and checkable, and they explained their reasoning better than we expected. The safety net matters more than the prediction: the bug that reached production was ours, in the rollback, not the agent's.

What's next for Pre-mortem-agents

• Check types for business logic (multi-step HTTP scenarios), so bugs like the oversold stock are covered.
• Pre-mortems for infrastructure changes.
• Earn autonomy: start Supervised and move to Hands-off as the measured hit rate rises.

Built With

• anthropic-claude
• artifact-registry
• cloud-build
• cloud-monitoring
• fastapi
• gitlab
• gitlab-ci
• gitlab-duo-agent-platform
• google-cloud-run
• mcp
• prometheus
• pytest
• python
• semgrep
• sqlite
• workload-identity-federation