Preflight

Inspiration

At my previous company, our customer support ran on Chatwoot, open-source software we hosted ourselves. Chatwoot has over 11 million Docker pulls, and most of those are self-hosted installs like ours.

In August 2026, version 4.17.1 shipped. A user followed the official upgrade steps, and every new conversation started failing with a 500 (chatwoot/chatwoot#16133). The fix was a restart. The upgrade guide never mentions one. A month earlier, 4.16.0 shipped a migration that drops an index without checking it exists. Self-hosters whose install lacked that index got a crash loop on boot and fixed it by hand.

CI caught neither. CI migrates an empty database and boots the app afterwards. Real installs are years old, have drifted, and keep running while they upgrade. Today that gap is covered by a maintainer reading every migration and guessing.

What it does

Preflight gives every merge request that touches the database a dress rehearsal, and a GitLab Duo flow fixes what the rehearsal catches.

• The rehearsal is a CI job that builds an aged Chatwoot install (two accounts, contacts, 200 conversations, around 700 messages), then replays the upgrade against it with real web and Sidekiq processes and live traffic:
  • Check 1, drift: for every object the migrations drop or rename, remove it from a copy first and migrate again.
  • Check 2a: old code keeps serving while the migration runs.
  • Check 2b: new code boots before the migration and keeps running after it, without a restart.
  • Check 2c: Chatwoot's documented Docker upgrade, run literally from the merge request's compose file.
  • Check 3: nothing left pending, whatever HTTP health says.
• The Migration Fixer is a custom GitLab Duo flow that starts when the rehearsal fails. It reads the failure, commits a fix to the merge request, and the rehearsal runs again. It may fix only two kinds of problem, gets two attempts, never edits the checks, never merges, and never decides about data.
• I approve the outcome. I review the fix and merge when the rehearsal is green. Merging builds an image, deploys it to Google Cloud migrating first, and a smoke check asks the running app whether any migration is pending.

Results on Chatwoot (live run, all public in the project):

For 4.17.1 the agent's fix was not just code. It added a one-shot migrate service to the compose file so the app starts only after the migration, and guarded the two column writes the rehearsal showed crashing. Where its own guard silently drops a write until restart, it said so in its note and told me what to check.

How I built it

• Verify: the migration-rehearsal job on every merge request that changes db/, app/models/, config/ or docker-compose*.yaml, on a docker:dind runner; plus the smoke check after every deploy.
• Create: the Migration Fixer flow, triggered by a failed pipeline, commits to the merge request.
• Package: the image is built and pushed to Google Artifact Registry.
• Configure: the deploy has no stored keys. GitLab's OIDC token is exchanged through Workload Identity Federation, limited to this project's protected branches. App secrets live in Secret Manager.
• Release: the demo environment on a Compute Engine VM. The deploy runs the migration in a one-off container first, then starts the app.
• Monitor: the smoke check needs Postgres and Redis healthy and asks the running app if any migration is pending.
• Govern: the rules in AGENTS.md. The checks always come from the target branch, so a merge request (or the agent) cannot weaken the checks that judge it. Two attempts, then a hand-off. Data decisions go to a person with the exact decision spelled out.

Seven of the nine stages. Not covered: Plan and Secure.

What I changed in Chatwoot

Chatwoot is upstream work (https://github.com/chatwoot/chatwoot, MIT), imported as v4.15.1 with the separately licensed enterprise/ folder removed (FORK.md). Upstream's v4.16.0 and v4.17.1 come in as one commit each, through merge requests. Everything else was added for this hackathon: the rehearsal toolkit (rehearsal/), the CI pipeline, the Migration Fixer flow and its rules, the Google Cloud setup and deploy (scripts/), and the fixes in MRs !5 and !6.

Challenges I ran into

• A stale schema only breaks records the app creates. Records loaded from the database still carry new columns, so a read-only health check passes. The rehearsal had to drive Sidekiq to create new records (an inbound SMS webhook) to see the bug.
• Chatwoot makes outbound calls (SMS provider, Gravatar), so the rehearsal runs on an internal Docker network to stay deterministic and offline.
• ActiveRecord::PreparedStatementCacheExpired shows up intermittently when a table gains a column under load. The rehearsal treats it as a warning only when the same Sidekiq job later succeeds on retry.
• The group's dependency proxy is off, so CI pulls images from Docker Hub; and the flow's runner image needed curl to start the Duo CLI.

Accomplishments I'm proud of

The rehearsal reproduced the exact error from the public bug report, undefined method 'status_changed_at=', in a merge request, which is where it would have been stopped before it shipped. And the agent didn't just patch code: it changed the order of the deploy, which is the real fix.

What I learned

The interesting bugs aren't in the migration. They're in the gap between the migration and the running app, and you only see them if you upgrade a running, aged install the way real users do.

What's next

Run Preflight on Chatwoot's upstream release branches, and extend it to other self-hosted Rails apps, whose upgrade guides follow the same pull, up, migrate pattern.

Built With

• artifact-registry
• bash
• docker
• docker-compose
• gitlab
• gitlab-ci
• gitlab-duo
• google-compute-engine
• postgresql
• python
• ruby-on-rails
• secret-manager
• sidekiq
• workload-identity-federation