CalmFix

• 

  CalmFix title, tagline and the 9-step loop (the thumbnail)

• 

  The alert becomes incident #5

• 

  Triage agent's root cause (commit 65388a7f, line 28)

• 

  The agent refuses an untested fix and escalates (the guardrail)

• 

  Its one-line fix plus regression test

• 

  Guardian approval (9/60 lines, all rules pass)

• 

  CI gates, including policy-gate, all green

• 

  Canary → verify → promote

• 

  Automatic rollback (p95 1347 ms vs 800 ms)

• 

  Release with INC-5.json audit log

• 

  The scribe's postmortem

• 

Inspiration

An alert fires at 1 AM. By morning the bad commit is found, a fix is merged and the canary is promoted, and nobody was paged. The one thing that makes that scary is how little anyone can prove about what the agent was allowed to do.

CalmFix is an incident-to-fix loop for a production service on Google Cloud Run. A policy file, .agent-policy.yml, sets what the agents may do alone. Every action is checked against it and recorded in a tamper-evident audit log.

What it does

1. A Cloud Monitoring alert (5xx ratio or p95 latency) calls a small bridge service.
2. The bridge opens a GitLab incident issue. It includes the failing revision, its git SHA, the top error messages and a stack trace. It then assigns the issue to the incident-triage flow.
3. The triage agent finds the suspect commit from the history and posts the root cause. It writes the smallest fix plus a regression test and opens an MR.
4. A guardian agent, running inside the same flow with its own prompt and a read-and-label toolset, reviews the MR against the policy. It approves, or labels it needs-human.
5. The MR pipeline runs tests, SAST, secret detection, dependency scanning, pip-audit, a severity gate and an independent policy gate.
6. A merge governor merges only an MR labelled agent::approved, with no needs-human label and a green pipeline for that exact commit.
7. Main deploys a 10% canary and sends it synthetic traffic. It promotes the canary, or rolls back on its own.
8. Every decision goes into a hash-chained audit log, attached to the GitLab Release. The release-scribe flow writes the postmortem on the issue.

The problem

Agents can already write plausible fixes. What keeps teams from letting them near production is the lack of hard limits and the lack of proof afterwards. An agent that clears an alert by deleting a test or editing the deploy config is worse than no agent at all.

How we use GitLab

• Duo Agent Platform: two enabled flows. incident-triage runs the triage agent and the guardian agent, and release-scribe writes the postmortem. They start on the Assign trigger.
• GitLab CI/CD: the security scanners, canary, verify, promote and rollback jobs, environments, and Releases with the audit log attached.
• Governance outside the agents' reach: the policy gate is defined on main, a path agents may not change. A merge governor does the merging, because agents have no merge tool. Workload Identity Federation means Google Cloud accepts only protected refs, so an agent branch can never deploy.

Stages covered

All nine.

How we built it

Python 3.12 with FastAPI, pytest, Docker, Terraform, Cloud Run, Artifact Registry and Cloud Monitoring. The orders service has demo bugs injected as real commits. Each one passes the committed test suite and fails its own reproduction test, so it is shaped like a real regression. make verify-bugs checks that.

Challenges

• Triggers fire only on human actions. A flow can't start another flow by assigning it. So the guardian runs as a second agent inside the triage flow.
• Bad commits that CI catches never reach production. So every demo bug passes the suite, as real regressions do.
• Not every bug shows up in a canary. The pagination bug fails only on the last page, so it passed the canary's smoke traffic and the production alert caught it. The N+1 latency bug failed the canary and was rolled back.
• Image scanning would block every deploy on base-image CVEs that have no fix. The image gate blocks only High or Critical findings that have a fix available, and it lists the rest.
• The agent sandbox had no package index. The first run escalated for exactly that reason. We made the sandbox setup best-effort and let CI run the tests.

Accomplishments that we're proud of

• Two real incidents ran with no human in the loop. Incident #5 came from a real Cloud Monitoring alert. The agent traced it to its commit and fixed it in MR !3. The guardian approved, the governor merged and the canary was promoted. A second bad change (the N+1 bug) failed its canary at p95 1347 ms against an 800 ms limit, rolled back on its own, and the agent then fixed it in MR !5.
• Governance that holds without trusting the agents. The policy gate lives on main, agents have no merge tool, and deploys are limited to protected refs.
• A verifiable audit trail. Each Release carries a hash-chained log (INC-5.json, INC-7.json). Anyone can re-check it with python scripts/audit_collect.py --verify INC-5.json.
• The agents stopped themselves when they should have. One run escalated to a human instead of guessing when its sandbox was broken.
• 58 passing tests, and every pipeline job under 5 minutes.

What we learned

• Hard limits matter more than better prompts. The agent's behaviour was never the safeguard. The policy gate, the governor and the cloud identity rules were.
• Make the demo bugs realistic. If a bug fails CI it never reaches production, so the interesting part of the story never happens.
• A canary only catches what its traffic exercises. Smoke traffic caught the latency regression and missed the last-page bug. That is why the production alert stays in the loop.
• Audit lines must be machine-checkable. Free-form agent comments weren't enough, so agents emit structured AUDIT-V1 lines that the pipeline validates and chains.
• Be honest when something is left open. Incident #5 stays open with postmortem::draft, because a human-flag label is still set and a person should sign off.

What's next for CalmFix

• A policy language with per-path budgets, and approval routing to code owners.
• Real SLO-based canary analysis from Cloud Monitoring, beyond smoke traffic.
• Using the standalone mr-guardian flow on human MRs as well.
• Letting a team tune what agents may do from one policy file, per service.

Built With

• agent
• artifact
• ci/cd
• cloud
• docker
• duo
• fastapi
• federation
• gitlab
• google
• identity
• manager
• monitoring
• platform
• pytest
• python
• registry
• run
• secret
• terraform
• workload