Rollback Safety Agent

Inspiration

You ship release N+1. It adds a refunded order status and starts writing it to the database. Twenty minutes later something unrelated breaks, so you do the safe thing and roll back to N.

N reads the orders table, hits status='refunded', and crashes:

ValueError: 'refunded' is not a valid OrderStatus

The rollback was supposed to be the safety net. Now it's the outage.

Teams test that a deploy works. Almost nobody tests that it can be undone once the new release has already written data. Tools like Liquibase, Atlas, and pgroll check whether the schema can be reversed, but a new enum value reverses cleanly at the schema level and still crashes the old app. I wanted something that tests what actually fails mid-incident: does the old application still work on data the new application already wrote?

What it does

Rollback Safety Agent proves rollback safety on every merge request. When the proof fails, GitLab Duo agents restructure the release until it passes, before anything merges.

1. A deterministic rollback prover (no AI). On every MR that touches app/ or migrations/, a CI job runs the last release (N) and the MR head (N+1) against a scratch Postgres in four steps:

Test	App	Database state	Question
baseline	N	N schema + seed	Does N work today?
forward	N+1	N+1 schema	Does the new release work?
app_rollback	N	N+1 schema + rows written by N+1	Can we roll back the code after N+1 has written data?
full_rollback	N	migrated back down	Does the downgrade work without losing data? (invariants: row counts, checksums of preserved columns)

The output is an evidence.json file with a verdict (SAFE/UNSAFE), a risk class, and every failing test. A red proof fails the pipeline.

2. The Rollback Guardian flow (GitLab Duo Agent Platform). When the proof fails, two agents run:

• Analyst reads the evidence and the MR diff, classifies the risk (enum_value_added, column_renamed, or unknown), and posts a diagnosis on the MR.
• Remediator rewrites the change into a safe multi-phase release:
  • For an enum: ship a tolerant reader first, with the new writes behind a flag.
  • For a rename: expand/contract with dual-write.

It commits the fix to the MR branch, which re-runs the proof. Then it posts a before/after matrix and opens a contract-phase issue with a due date.

3. Release with proof attached. Every merge creates a GitLab Release with the verdict and evidence.json attached. Production deploys need manual approval. After deploy, a health check runs, and if it fails, the pipeline automatically rolls back to the proven revision.

Autonomy: Supervised. The agents diagnose, fix, and re-prove on their own. A human approves the merge and the production deploy, and the agents never do either. An unknown risk class gets no fix attempt and goes straight to a human, with the matrix and the failing tests.

How we built it

• Demo app: FastAPI, SQLAlchemy, Alembic, and Postgres, with a native enum.
  • /health is a deep check that hits the database.
  • /livez is a shallow liveness check.
• Contract tests: 11 HTTP-only tests driven by BASE_URL, safe against any database state. Because of this, any version's tests can run against any version of the running app, which is what makes the version × data-state matrix possible.
• Prover: a Python CLI that runs N and N+1 from git worktrees as processes. It returns exit code 0 (SAFE), 1 (UNSAFE), or 2 (harness error) and has 21 unit tests of its own.
• CI/CD: image built by digest, contract tests, SAST, secret detection, container and dependency scanning, rollback proof, release, manual-approval deploy, health check, and auto-rollback.
• Agents: a Duo custom flow (flows/rollback-guardian.yml) plus a pattern library (flows/patterns.md). For each risk class, the library gives the signal, the safe rewrite, an example diff, and the contract-phase task.
• Guardrails: the Remediator may edit only app/ and migrations/. It can never touch tests/ or prover/ and can't delete files. It gets at most 2 attempts, enforced by a CI guard job, and CODEOWNERS requires a human review.
• Runtime: Render (app) and Neon (Postgres), deployed and rolled back from GitLab CI through Render's API.

DevSecOps stage	Implementation
Plan	Agent opens a contract-phase issue with a due date
Create	Remediator commits the safe rewrite
Verify	Contract tests and the rollback proof matrix
Secure	SAST, secret detection, container and dependency scanning; remediator path guard
Package	Immutable images by digest
Release	GitLab Release with the proof attached
Deploy	Manual-approval deploy, migrations first
Monitor	Health check and auto-rollback to the proven revision
Govern	CODEOWNERS, guard job, evidence as audit trail

Challenges we ran into

• Two app versions in one CI job. Docker-in-Docker on hosted runners was a risk, so the prover runs both versions from git worktrees against a scratch database.
• Tests that tolerate any data. The contract tests had to be rewritten so they never assume a clean database. Otherwise app_rollback fails for the wrong reason.
• Stopping the agent from cheating. The easiest way to turn a check green is to edit the check. The path guard and CODEOWNERS make that impossible rather than unlikely.
• Expand/contract spans releases. Getting the agent to split a rename instead of patching it in one commit meant encoding the contract phase as a tracked issue.
• Permissions. With Developer access, I can't enforce "Pipelines must succeed" on main myself. The settings are documented in docs/merge-gate.md for a Maintainer.

Accomplishments that we're proud of

• Enum scenario: 5/5 runs went from red proof to agent fix to green proof on the first attempt.
• Rename scenario: 5/5 runs went green on the first attempt.
• Unknown risk class: the handoff to a human was verified, with no fix attempted.
• Phase 2 of the enum release (turning on the new writes) also proves SAFE.
• Auto-rollback was demonstrated in production: pipeline 2983342588.
• A complete loop: proof → diagnosis → fix → re-proof → release with evidence → deploy → auto-rollback.

Honest caveat: patterns.md contains near-exact example fixes for these two risk classes. These numbers show the agents reliably applying known safe patterns, not generalizing to arbitrary changes. The retry path (attempt 2) was never triggered, so it is untested.

What we learned

• Most rollback outages are about data shape, not schema. The schema can roll back cleanly while the app still breaks.
• Agents are most dependable when the hard judgment is deterministic. The prover decides whether a release is safe, and the agents choose from a fixed set of safe rewrites. The same proof then checks their output.
• A proof is only as strong as its contract tests, so the tests became the most important part of the project.

What's next for Rollback Safety Agent

• More risk classes, starting with NOT NULL columns without a default.
• A mixed-fleet test (N and N+1 running concurrently) and canary traffic splitting.
• Proofs against anonymized snapshots of production data instead of seed data.
• An enforced merge gate. Today it is documented but needs a Maintainer to turn on.

Current limits: a SAFE verdict covers the supported risk classes and the seeded data only. It's Postgres only, and anything outside the two classes goes to a human.

Built With

• alembic
• docker
• fastapi
• gitlab
• gitlab-ci
• gitlab-duo
• gitlab-duo-agent-platform
• postgresql
• pytest
• python
• render
• sqlalchemy