Falsify

GitLab Duo code review that shows a reproducible failure, lets you approve the fix, and keeps the accepted test to catch regressions. No red test, no accusation.

Comment

Inspiration

What should an AI code review leave behind? A test that catches the bug next time.

For a solo developer or small team, a review comment is often the beginning of another investigation. What actually breaks? Is the proposed fix correct? Will the same mistake return in the next change?

We built Falsify around that last question. GitLab Duo proposes a test, CI checks the evidence, and a human decides what to accept. When the fix is accepted, the exact test becomes part of the repository.

No red test, no accusation.

The moment that explains Falsify

We deliberately introduced a Unicode bug: text such as “Café Ñandú 日本語 ЖЯ” lost its non-ASCII characters. It is a small code change with an immediately understandable consequence: someone's name or language gets damaged.

Duo proposed a test. After human approval, Falsify ran it three times: twice against the changed code, once against the earlier version. It failed twice on the change and passed on the earlier version. Certification stopped.

A human accepted the fix. The same test passed and was retained, byte for byte, as a permanent regression test.

Then we deliberately put the bug back on an isolated branch.

The retained test caught it again. The useful result of the review survived after the conversation ended.

This was a deliberately seeded demonstration. Separately, an unseeded review of our application found two repeatable crashes in deeply nested JSON records. Both were fixed through Duo-proposed suggestions and retained as tests. Those were failures against new-code requirements, labelled separately because the target did not exist in the earlier version.

Follow the Unicode proof · Inspect the unseeded review

How it works

Falsify connects three custom GitLab Duo Agent Platform flows—Plan, Trial and Monitor—with GitLab CI/CD:

  1. Plan: turn an issue into testable claims, then pause for human approval.
  2. Trial: propose focused pytest experiments, pause for approval of the exact source, and let CI execute them in isolated Docker sandboxes.
  3. Resolve and retain: accept a remedy or challenge the test's interpretation. Accepted fixes require a passing rerun and retention of the exact test before certification.
  4. Release and observe: the demonstrated installation carries certified changes through staging, explicit production approval, Cloud Run deployment and a ten-minute watch. The Monitor flow can explain watch evidence when a human requests it.

A reproducible test can still demand the wrong behavior. In a separate owner-authored control, a test demanded zero where the approved requirement said None. A human upheld the challenge; certification recorded a strike against Falsify and excluded the test from promotion. The reviewer has to be accountable too.

What we built and verified

The application uses Python, FastAPI and pytest, with Docker for experiment isolation and Google Cloud Run for deployment. GitLab records the proposals, approvals, results and release evidence. AI proposes; named humans authorize; CI executes and checks.

At the October 14 release checkpoint:

  • 419 tests passed, with three explicit integration skips. Three real Docker controls passed separately in native CI.
  • 6 of 6 approved staging checks passed, covering health, feature flags and representative eligibility behavior.
  • The production watch replayed 11,964 requests over ten minutes with zero observed replay failures. These figures describe that bounded service check; they do not measure AI review accuracy. A separate rollback drill is linked in the evidence.

Inspect the release pipeline · Inspect the exact production watch

Challenges and lessons

The hardest part was keeping evidence attached to the code it actually tested. Change the code, test or runtime, and an old approval may no longer apply. Falsify binds those decisions to exact versions and refuses stale evidence.

Our own staging check also caught a health endpoint that worked locally but returned 404 on Cloud Run. It stopped the release until we corrected the path. That made the value of the release checks concrete during development.

The biggest lesson: executing a test and deciding whether its expectation is correct are separate responsibilities. Falsify makes room for both.

Try it, and what's next

Explore the live project or run the local rehearsal to see real Docker execution, test retention and the returning regression. The rehearsal uses scripted proposals and decisions; the linked native GitLab records document the historical Duo workflow.

Developer + AI compatibility is the next installation priority and is not implemented yet. The historical native releases used Maintainer access and protected branches/environments. The organizer has said that access may not have been intended and instructed us not to rely on privileges beyond Developer + AI; it is not an approved exception. The local rehearsal runs without those GitLab privileges. Falsify currently targets Python/pytest; the hosted demo uses ephemeral SQLite.

After validating the supported installation path, we want to package it as a reusable CI/CD component so more teams can turn an accepted review finding into a lasting test.

Built with

GitLab Duo Agent Platform, GitLab CI/CD, Python, FastAPI, pytest, Docker, Google Cloud Run, Artifact Registry, Workload Identity Federation, Secret Manager and SQLite.

Project links

Built With

  • artifact-registry
  • docker
  • fastapi
  • gitlab-ci/cd
  • gitlab-duo-agent-platform
  • google-cloud-run
  • pytest
  • python
  • secret-manager
  • sqlite
  • workload-identity-federation
Share this project:

Updates

Submission history