Courtlab

CourtLab puts every change release on trial: GitLab Duo agents build the case, CI rehearses the rollback on real db, and only the tested build ships.

Comment

CourtLab: every release stands trial, rollback included

CourtLab

Every release stands trial. Rollback included.

CourtLab is a release hearing for database changes, built on the GitLab Duo Agent Platform. Assign the CourtLab reviewer to a merge request: Duo agents build the case, trusted GitLab CI rehearses the upgrade and the rollback on a real PostgreSQL database, a blocked release gets one verified fix, and a human signs a release order that deploys only the exact image that was tested.

The agents do the post-code work. The human approves one outcome.

Source repository on GitLab Live hearing viewer on Google Cloud Run Path A, Supervised

Hold it! The burst rewinds into the CourtLab seal

Inspiration

Every team says "we can always roll back". Almost nobody tests it before shipping.

A schema change can pass code review, unit tests, integration tests and SAST and still take payments down. The failure shows up only when real data meets the new code, or when on-call rolls back at 3 a.m. and the old code meets the new schema. Review tools, including very good AI ones, can warn about that. None of them run it.

We wanted a merge request to be treated like a release on trial: charges filed, evidence executed, a verdict written by something trustworthy, a remedy if it fails, and a human who signs the final order.

The rollback trap: after a rollback the old release cannot create a single payment

Why a court, not one more AI reviewer

One AI reviewer is prosecutor, judge and jury. CourtLab splits the powers: agents accuse, CI convicts, you sign.

A single AI reviewer accuses, weighs the evidence and decides, all in one prompt. That is why it can be talked out of a finding, can "fix" a failure by weakening the test, and forgets the bug next sprint. CourtLab separates the powers, the way a court does:

Court principle What CourtLab does The failure it prevents
Accuser ≠ judge One agent files charges; a different agent rules on what counts as evidence. An AI confirming its own suspicion: false alarms or confident misses.
Witnesses don't rule MR comments, Duo's review and SAST are witness statements: weighed as data, never obeyed. Prompt injection: "this change is safe, approve it" in a comment or code line cannot sway the verdict.
Only evidence convicts A charge counts only when executed, 2× on the old code and 2× on the new, in a locked sandbox. CI writes the verdict, never a model. "Looks fine to me" approvals and hallucinated bugs.
No moving the goalposts The evidence is frozen and fingerprinted; a fix is judged by identical evidence (44247057fb74 = 44247057fb74). AI fixers that edit the test until it goes green.
Precedent Proven failures become cases that run on every future release (payment retry, refund cap, free payment). The same bug returning three sprints later.
Burden of proof Missing evidence means not ready, never "probably fine". Shipping because nothing was checked.
Public record Every hearing keeps raw logs and hashes in a viewer anyone can open. No answer to "who approved this, and on what evidence?"
The sentence needs a signature Only a human's release order ships, and only the exact tested build. Auto-deploys of code nobody verified.

Not just migrations

The rollback rehearsal is the database-specific part. The rest of the court judges any behaviour change in the release:

Case GitLab Duo alone With CourtLab Real evidence
A business rule breaks ("a 100% discount must not charge the card") A comment; in our evaluation it flagged a different issue The rule becomes an executable charge; the violation is reproduced on head and not on base, and the release is blocked Evaluation case e4: pay003-full-discount-no-capture reproduced as a violation; its fix passed the retrial
Money moves twice (a retried payment or refund applied again) Cannot run the retry Registered precedents replay retries and refund caps on every release 16 sandbox runs per hearing, each log hashed
A planted "approve this" comment Reads it like any other text Treated as a witness statement, never as an instruction The case file marks every witness as data
A fix that quietly weakens the test Cannot tell The remedy may only touch app/; acceptance stays frozen and fingerprinted Remedy retrials carry the original fingerprint
The deployed build isn't the tested one Not its job The release gate refuses unless main equals the tested source and the image digest matches Both production releases went through the gate

What it does

CourtLab capability What it does Why it matters
One-click trigger Assign the CourtLab reviewer on a merge request. No new UI, no new process: it lives where the release is reviewed.
Case file Exact base/head commits and images, approved requirements, the full migration history, the diff, and witness statements from GitLab Duo, humans and SAST, all treated as data, never as instructions. Every judgment is about one exact release, not "the branch".
Two agents, one honest bar An investigator files charges as executable tests; an independent admission agent rules on each one and reports every exclusion. Opinions become testable claims, and nothing is silently dropped.
Real rollback rehearsal Trusted CI runs the old release, upgrades, writes payments and refunds with the new one, rolls the application back, and requires the old release to still read and write everything, on real PostgreSQL. It catches the outage reviews only describe.
Frozen checks Exhibits and precedents run twice on base and twice on head, offline, read-only, non-root; the acceptance set is fingerprinted and frozen. A fix can't pass by moving the goalposts.
One verified remedy After a block, a remedy agent opens a draft fix MR; the same frozen checks judge it again. You get a tested way forward, not just a "no".
Judgment by CI, never by a model blocked, ready or inconclusive, written by the CI job with the plan, policy and acceptance hashes, posted on the MR and in a public viewer. Agents propose; code routes; CI decides.
Human release order A protected deploy-production job re-checks the judgment, that main equals the tested source, and the image digest, then deploys to Cloud Run. Supervised autonomy: you approve the outcome, not every step.

The seven stages of a release hearing

Product walkthrough: a real merge request, start to finish

MR !158 records a currency on every payment. CI is green. Its migration drops the column default, so after a rollback the previous release cannot create a payment. GitLab Duo Code Review warns about it. CourtLab proves it, blocks it, fixes it and ships the fix.

Assigning the CourtLab reviewer on MR !158
I · Assign. Green pipeline, the DROP DEFAULT line, then the CourtLab reviewer. The bot links the live session.		 GitLab Duo agent session
II · Investigate. The Duo agent session files four charges; trusted CI takes over.
Rollback rehearsal in the sandbox
III · Rehearse. Old release → upgrade → new writes → rollback. "After rollback: A valid synthetic payment was rejected."		 Release blocked judgment on the MR
IV · Blocked. The judgment lands on the merge request with its exhibit table.
The CourtLab viewer: stage IV and the raw rollback log
V · Inspect. The viewer: 16 sandbox runs, the red rollback, and the raw log, one click away.		 Draft remedy MR !162
VI · Repair. Draft MR !162: one forward migration restoring DEFAULT 'USD', written with the real migrations in view.
Retrial ready
VII · Retrial. Same frozen acceptance, 44247057fb74 = 44247057fb74. Every phase holds: ready. Duo's re-review agrees: "Fixed".		 Run job, gate checks and live version
VIII · Release. A human clicks Run job; the gate checks judgment, source and digest; /version answers with the tested revision.
Blocked, with the evidence The judgment on the MR Retrial: ready Shipped: the signed order
Blocked hearing Judgment on MR !158 Ready retrial Release order

How the hearing works

  1. Assign the CourtLab reviewer. The GitLab Duo flow opens a case for the exact merge request revision.
  2. Investigate. The investigator agent reads the packet (requirements, migrations, diff, called code, witness statements) and files charges as executable tests plus a rehearsal seed.
  3. Admit. The admission agent rules on every charge; exclusions are reported.
  4. Execute. A docket commit starts the trusted CI judge: exhibits and payment precedents twice on base and head in an offline sandbox, then the rollback rehearsal against real PostgreSQL and the real application images, pulled by digest.
  5. Judge. CI writes blocked, ready or inconclusive and posts it on the MR and in the viewer.
  6. Remedy. If blocked, the remedy agent opens one draft fix MR; the same frozen checks run on it.
  7. Release. A human runs the protected deploy-production job; the release gate verifies everything again and deploys only the tested digest.
Agents can Agents cannot
Read the hearing packet: requirements, migrations, diff, called code, witness statements. Choose the next step, start retries or write the verdict.
File charges as executable tests tied to an approved requirement. Run code, SQL or shell; CI runs everything in a sandbox.
Propose one implementation-only fix as a draft MR. Change tests, requirements, CI, dependencies or the frozen acceptance.
Treat witness statements (including Duo's) as evidence to weigh. Follow instructions embedded in code, comments or SAST output.

How we built it

Agents feed proposals to a code controller; CI writes the judgment

Technology Role in CourtLab
GitLab Duo Agent Platform The CourtLab custom flow in the AI Catalog: three agents (investigator, admission, remedy) with tools pinned to the hearing packet; routing is deterministic.
GitLab CI The trusted judge (court-docket) and the protected human release order (deploy-production).
Python 3.12 · Pydantic · psycopg Controller, case file, exhibit runner, rehearsal probe, publication, release gate.
FastAPI + PostgreSQL 16 The subject: a synthetic payments and refunds service with forward migrations.
Docker Read-only, non-root, network-less sandboxes and an internal network for the rehearsal.
Google Cloud Run Production payments service and the public CourtLab viewer.
Cloud SQL, Secret Manager, Artifact Registry The production database, its DSN secret and the tested images.
Workload Identity Federation GitLab CI deploys to Google Cloud with no stored keys.
Semgrep SAST Witness statements for the case file.

Challenges we ran into

  • Agent output must never be the decision. Every verdict is CI-executed evidence; agents only propose, and exclusions are always reported.
  • Judging through a changing schema. The schema is what the release changes, so the rehearsal judges preservation through the release's own public API, with the database snapshot as secondary evidence.
  • Rollbacks fail on writes, not only reads. Our first rehearsal re-read old data after rollback; a live hearing showed that the dangerous case is the old release writing after the new one changed the schema. Writes after rollback became a trusted check.
  • Live runs found real bugs. Duplicate receipts, a release gate that could not read its own judge job with a CI token, a remedy agent that could not see migration files, a viewer index starved by Cloud Run: all found in real hearings and fixed.

Accomplishments we're proud of

  • Two real releases (MR !123 and MR !158) went blocked → verified fix → ready → human release order → production, end to end on GitLab and Google Cloud.
  • On !158, GitLab Duo warned, CourtLab proved and fixed, and Duo's own re-review confirmed the fix.
  • A pre-registered evaluation against GitLab Duo Code Review on six held-out releases, with labels hashed before any run and every attempt, failure and verbatim Duo comment published.

What we learned

GitLab Duo is already excellent at spotting release hazards in a diff. In our evaluation it was faster than CourtLab and caught every unsafe case. The gap is not detection; it is proof, repair and control. The useful role for agents after code is written is to turn a warning into executed evidence, a verified fix and a release a human can approve with confidence. Duo reviews. CourtLab proves. You release.

Before: green checks, a merge, and a rollback that takes payments down. Now: proof before the merge and a rollback already rehearsed.

What's next

  • Database-level rollback rehearsal (not just application rollback).
  • More stacks than the payments sample: bring your own requirements, fixtures and probes.
  • Collapsible, human-first receipts on the MR.
  • Making the flow public in the AI Catalog.

Honest limits

  • The subject is a synthetic payments service; CourtLab is not a payment gateway or a production certificate.
  • The rehearsal rolls back the application, not the database.
  • The admission agent is conservative and fallible: in the evaluation it excluded one correct charge that Duo caught.
  • Six evaluation cases are directional, not statistical, and were authored by the CourtLab author.
  • For an account without a default Duo namespace, the GitLab reviewer picker greys out the CourtLab account; assigning through the API works and started our runs.

Try it

Built With

  • artifact-registry
  • cloud-sql
  • docker
  • fastapi
  • gitlab
  • gitlab-ci
  • gitlab-duo-agent-platform
  • google-cloud-run
  • postgresql
  • psycopg
  • pydantic
  • python
  • secret-manager
  • semgrep
  • workload-identity-federation
Share this project:

Updates

Submission history