Skewcheck: migration gate with a GitLab Duo repair agent

Skewcheck: the agent proposes, the gate decides, a human merges. Real snapshots: blocked, repaired by GitLab Duo, verified
The problem: in a rollout, v1 and v2 share one database. A migration for v2 can pass every check and still break v1
MR !4: shop v2 renames customers.email to email_address. v2's own tests pass. It looks harmless
The gate runs v1's tests on the new schema: SQLSTATE 42703 in both runs, never on the old one. UNSAFE, pipeline blocked
skewcheck-repair: a real GitLab Duo Agent Platform flow in the AI Catalog, enabled in the project with a mention trigger
v1 passes on today's schema, fails on the proposed one. A fixed rule links the error to the removed column. No AI decides
One comment starts the GitLab Duo flow. It reads the gate's evidence and opens repair MR !5, marked verification pending
The agent's fix is expand/contract: keep email, add email_address, sync both with a trigger. The drop waits for v1 to retire
The same gate judges the repair, using the engine from the target branch: SAFE for v1, v2 still passes. A human merges
Full lifecycle on main: CI image, tests on real PostgreSQL, the gate, keyless deploy to Google Cloud Run, live smoke test
Live on Google Cloud Run: 11 scenarios, five honest verdicts, each re-runnable on a fresh PostgreSQL in about 4 seconds
Footprint measured with SCI (ISO/IEC 21031): 0.081 gCO2e per gate run, 84% less runner time on MRs that can't change the verdict
verifier-guard fails any repair branch that edits the engine, tests, policy, CI, Duo flow or baseline. No weakening its judge

Inspiration

Every team that runs a rolling deploy has a short, dangerous window: the new schema is live, but the old version of the app is still serving traffic. A migration written for v2 (renaming a column, tightening a type, adding a CHECK) can pass code review, pass the linter, pass v2's own tests... and still take down v1 the moment it runs.

The usual answer is "follow expand/contract and be careful." But nothing in a normal pipeline actually runs v1 against v2's schema, so nothing proves you were careful. GitLab's own Gitaly team once tried running old-version tests against newer code in CI and dropped it when the old suite went stale. That told me the idea is valuable, and that the hard part is evidence quality: knowing when a failure is a real incompatibility rather than a stale assertion, a flaky test, or broken infrastructure.

I also wanted an agent that does real post-code work without being trusted blindly. A migration break is a perfect case: the fix (expand/contract) is well known but tedious, and whether a fix works can be checked mechanically. So the design principle became:

The agent proposes. The gate decides. A human merges.

What it does

Skewcheck is a merge-request gate for PostgreSQL migrations, with a GitLab Duo Agent Platform repair flow.

A developer opens a merge request with a migration. In the demo, shop v2 renames customers.email to email_address; v2's own tests pass.
The skewcheck-gate CI job starts a throwaway PostgreSQL 16 and runs a controlled experiment:
- control: previous app version (pinned git tag) on the baseline schema
- candidate, twice: previous app version on the proposed schema (the actual compatibility question)
- context: current app on the proposed schema
It returns one of five verdicts (SAFE, UNSAFE, ADVISORY, NO_EVIDENCE, INCONCLUSIVE) as JSON, Markdown and JUnit reports, and blocks the pipeline only on proven problems. In the demo: UNSAFE, SQLSTATE 42703 column "email" does not exist, reproduced in both runs and absent on the old schema.
The developer mentions the skewcheck-repair Duo flow on the MR. The agent reads the gate's machine-readable evidence from the job log, reads the migration and config, writes an expand/contract repair (keep email, add email_address, sync both with a trigger, defer the drop), and opens a repair MR into the developer's branch. Its description quotes the evidence and states "verification pending".
The same gate judges the repair, with the engine and policy compiled from the target branch so the repair can't weaken them, plus a repair-acceptance check so "just delete the migration" isn't accepted. A verifier guard fails any repair branch that touches the engine, tests, policy, CI or the baseline pin. Result: SAFE, v2 still passes, and a human merges.
On main, GitLab runs the rest of the lifecycle: tests, the gate, a keyless deploy to Google Cloud Run (Workload Identity Federation, no service-account keys), and a smoke test of the live URL against the deployed commit.

The verdict is deterministic. A test $t$ proves a regression only if it passes in the control run, fails in every candidate run, and shows a new database error $e$ that a fixed rule $\rho$ links to a changed schema object $\delta$:

$$ \text{UNSAFE} \iff \exists t, e : \quad \text{pass}^{\text{ctl}}(t) \wedge \text{fail}^{\text{all}}(t) \wedge e \in E^{\text{all}}(t) \setminus E^{\text{ctl}}(t) \wedge \rho(e, \delta) = \text{strong} $$

Here $E^{\text{all}}(t)$ is the set of normalized errors (SQLSTATE + message + statement fingerprint) that test $t$ produced in every candidate run, and $E^{\text{ctl}}(t)$ is the same set for the control run. There are six rules, for example 42703 undefined_column with a removed column, 23502 not_null_violation with a new NOT NULL, and 23514 with an added CHECK. No model is involved in any verdict. Anything weaker becomes ADVISORY; anything that didn't run cleanly becomes INCONCLUSIVE, never a pass.

Skewcheck report

Try it: the live Cloud Run demo lets anyone re-run any of the 11 scenarios against a fresh PostgreSQL in about 4 seconds. The real loop is on GitLab: !4 (blocked) and !5 (agent repair, verified).

How we built it

Engine (TypeScript, Node 22).

Its own throwaway PostgreSQL 16 cluster per experiment, with server-side JSON logging of every statement and error. Each test file gets its own application_name, so evidence is attributed per test whatever language or driver the app uses.
Schema snapshots straight from pg_catalog (columns, types, nullability, defaults, constraints, indexes, triggers), diffed into typed changes with a risk class.
Differential error analysis across control and repeated candidate runs, the six correlation rules, identifier-level coverage of risky changes (with static references in the old source for untested ones), and a pure verdict function with a documented policy and exit codes.
39 tests on real PostgreSQL: 11 scenario fixtures (safe, breaking rename, assertion drift, untested drop, missing baseline, broken migration, pre-existing failure, defective repair, valid repair, enum change, new CHECK, table rename), git-mode baseline tests, guard tests, and adversarial unit tests of the verdict logic.

GitLab CI/CD (the system of record).

Pipeline: ci-image (kaniko builds a Node + PostgreSQL image, only when its definition changes), then typecheck and engine-tests, then skewcheck-gate and verifier-guard, then example-reports, then deploy-cloud-run and deployment-smoke-test.
Merge-request jobs are change-scoped: the gate runs only when migrations, the app, the config or the engine change.
Reports are published as artifacts, as a JUnit test report in the MR, and as a JSON evidence block in the job log for the agent.

GitLab Duo Agent Platform.

A custom flow (flow registry v1, ambient) published in the AI Catalog and enabled in the project with a mention trigger.
It uses get_pipeline_errors, get_job_logs, get_repository_file, create_branch, create_commit, create_merge_request and create_merge_request_note.
Its prompt encodes hard rules: act only on UNSAFE, never touch protected paths, never edit an applied migration, use expand/contract, open the MR into the feature branch, never call its own change safe, and don't loop on its own repair branches.

Google Cloud. Cloud Run (us-central1, scale to zero, max 2 instances) serves the report viewer and runs live scenarios with the real engine inside the container. GitLab deploys it on every push to main through Workload Identity Federation; /health reports the exact deployed commit.

Sustainability: measured, not claimed

A gate that runs on every MR should be cheap, so I measured it with the Software Carbon Intensity specification (ISO/IEC 21031:2024) using real job durations from GitLab and published coefficients:

Runner: GitLab saas-linux-small-amd64, 2 vCPU / 8 GB, us-east1.
Power: Cloud Carbon Footprint GCP coefficients, 1.34 to 4.98 W per vCPU at 50% utilization, 0.392 W per GB, PUE 1.1.
Grid: Google's 2025 intensity for us-east1, 579 gCO2e/kWh.
Embodied emissions: amortized by time and vCPU share.

$$ \text{SCI} = \frac{E \cdot I + M}{R} \qquad P = \big( 2 \times (1.34 + 0.5 \times 3.64) + 8 \times 0.392 \big) \times 1.1 \approx 10.4 \text{ W} $$

$$ \text{Gate run, 41 s:} \quad E \approx 0.117 \text{ Wh} \quad E \cdot I \approx 0.069 \text{ g} \quad M \approx 0.012 \text{ g} \quad \text{SCI} \approx 0.081 \text{ gCO2e} $$

Measured on this project's own pipelines:

PostgreSQL baked into a CI image instead of an apt install per job: gate 60 s to 41 s, 0.118 g to 0.081 g per run (−32%); engine tests 87 s to 63 s (−28%).
Change-scoped MR jobs, measured on a docs-only MR: 4 jobs, 216 s, 0.426 g down to 1 job, 35 s, 0.069 g (−84%).
No model on the hot path: Google reports about 0.24 Wh for a median Gemini text prompt, more than an entire optimized gate run (about 0.12 Wh). The agent runs only after a proven UNSAFE result and a person's request.
Tuned ephemeral database (fsync off, nothing running between pipelines) and Cloud Run scaling to zero in us-central1 (88% carbon-free energy).

scripts/sci.mjs reproduces every number; the assumptions and limits are in docs/sustainability.md.

Challenges we ran into

Telling a real break from noise. Old tests fail for many reasons. I separated a regression (strongly correlated, reproduced in every run) from assertion drift (a test fails with no new database error, e.g. a changed default), pre-existing failures (already failing on the baseline), flakes (only some runs fail), and coverage gaps (no test touched the changed column, which gives NO_EVIDENCE, not SAFE).
Avoiding the stale-baseline trap. The baseline is an explicit, reviewable git tag recorded in every report. Editing an already-applied migration is detected, a mostly-failing baseline is INCONCLUSIVE, and only humans can move the pin; the guard rejects it on agent branches.
My own gate passed when it should have failed. On the first real MR, Skewcheck computed UNSAFE (exit 1), but the job went green: GitLab echoes each script line before running it, so a separate GATE_EXIT=$? line captured the echo's exit status. I caught it because the pipeline history didn't match the report, then fixed it by capturing the code on the same line.
Keeping the verifier honest against the agent. The CI file comes from the branch under test, so the engine and policy are compiled from the MR's diff base instead, and the guard blocks protected paths and baseline changes on repair branches.
Duo triggers. Flow triggers fire only on a person's action, not from CI. So the workflow is honestly Supervised: one human comment starts the agent, and a human approves the outcome.
Cloud details. New projects need build-account roles, Cloud Run reserves /healthz (moved to /health), and the CI deploy uses keyless federation restricted to this project's protected main branch.

Accomplishments that we're proud of

A real agent loop on gitlab.com, done twice (!1 to !2 and !4 to !5): the gate blocks, the Duo flow repairs from the evidence in about 1.5 minutes, the same gate verifies, and a human merges.
A verifier that can't be talked out of its verdict: deterministic rules, trusted-branch engine, guard, repair acceptance. A defective "repair" (adding email_address NOT NULL) is rejected with SQLSTATE 23502.
Five honest verdicts. Missing evidence and broken infrastructure never turn green.
The whole post-code lifecycle in one pipeline: CI image build, typecheck, tests on real PostgreSQL, compatibility gate, integrity guard, reports, agentic repair, Cloud Run deployment and live smoke test.
A measured footprint: 0.081 gCO2e per gate run, and 84% less runner time for MRs that can't change the verdict.

What we learned

Agents are most useful where the result can be checked mechanically. Pairing a generative step with a deterministic judge made the agent trustworthy without making it timid.
In CI, evidence quality matters more than detection. Repeated runs, a control run and explicit "I don't know" verdicts matter as much as catching the bug.
GitLab Duo Agent Platform flows are real infrastructure: YAML in the AI Catalog, service accounts, triggers and tool permissions. The trust boundary has to be designed around what the agent can touch.
Measuring with SCI changed my design: the biggest saving wasn't in my engine (which runs in about 3 seconds), it was in not running jobs that can't change the answer.

What's next for Skewcheck: migration gate with a GitLab Duo repair agent

More correlation rules and migration shapes (views, functions, partitions, ON CONFLICT targets), plus lock-time and table-rewrite warnings.
A contract-phase flow: when the baseline tag moves past v1, the agent proposes the deferred DROP COLUMN, and the gate verifies it against the new baseline.
Baselines taken from real deployment history (the oldest version still deployed) instead of a hand-set tag.
Ready-made adapters for common test runners (pytest, RSpec, Go) and ORMs, published as a reusable GitLab CI/CD component.
A merge-request widget showing the control/candidate matrix inline.

Links: GitLab repository · Live demo on Cloud Run · Duo flow in the AI Catalog · Demo video