Why this exists

A red pipeline gets a retry. The person who clicks it does not read the log. Two failures leave in the same red build. One is a flake. One is a real bug. The retry paints both of them green, and the bug ships.

That is the habit Flake Court is built for. Not a dashboard of flaky tests. Not a bot that invents a patch from a traceback. A court that separates the two before anyone is allowed to call the pipeline clean.

What we built

Flake Court is a GitLab CI court plus a published GitLab Duo flow.

The suite runs the tests. When the suite fails, the court job starts. It reads the junit file, takes every failed pytest node, and runs that node again in a clean process. Nothing from the previous test is still in memory.

Exit 0 is a flake. The court may place that node in quarantine.json for seven days, and that file is the only file it may commit. Any other exit is a product bug. The node goes on the leave-red list. The flow is forbidden to quarantine it. The pipeline stays red. A person merges. The flow does not edit application code, tests, or CI, and it does not press merge.

A guard runs on the next pipeline. It reruns every quarantined test alone. It rejects an entry that is expired, dated more than seven days out, missing an evidence URL, or still failing alone. A flake earns seven days. It does not earn a permanent skip.

How the two failures are split

The sample product bug is a price. A delivery of 10 kilograms over 3 kilometers must cost 1050 cents: 500 for the base, 300 for distance, 250 for the heavy surcharge. The spec says the surcharge starts at 10 kilograms. The program adds 250 cents only above 10. python -m app 10 3 prints 800. The same assertion fails alone and fails in the suite. The court marks it product. It stays red. Quarantine cannot hide it.

The sample flake is an id. tests/test_a_seed_id.py stores job-1. tests/test_z_fresh_id.py::test_fresh_job_1_in_a_clean_process expects a fresh id and finds job-1 still there, so the suite raises "duplicate id job-1". Run that same test alone, in a clean process, and it passes. The court marks it flake. It can be skipped for seven days, with the job URL as evidence, and then it is tested again.

The court log on the proof pipeline is one line: flake=1 product=1 report=court-report.json.

How the flow runs

The flow is published in the GitLab AI Catalog and enabled on the project.

https://gitlab.com/flake-builder/flake-court/-/automate/flows/1020115

One trigger listens for a pipeline event whose status is failed and whose source is a merge request. A person opens the merge request. The service account ai-flake-court-flake-builder runs the flow. The flow stops unless the court job succeeded and the report names a flake or a product bug. It may commit only the next quarantine list. It must not add a leave-red node. It posts one note on the merge request.

The repository is public, MIT licensed, at https://gitlab.com/flake-builder/flake-court

The proof pipeline is https://gitlab.com/flake-builder/flake-court/-/pipelines/2992846978

The demo is the real suite log and the real court log: https://youtu.be/dyZqaGLFQTA