FlakeProof

A GitLab Duo agent fixes flaky tests, but never judges its own fix: a CI proof job reruns the tests thousands of times and merges only on statistical proof.

Comment

Inspiration

Excalidraw, an open-source whiteboard with 134k stars, has this line in its test suite:

// FIXME too flaky. No one knows why.
it.skip("should bump the version of a labeled arrow when the label is updated", ...

Every large codebase has tests like this. Google reported that about 16% of its tests show some flakiness and 1.5% of all test runs are flaky. They pass and fail on the same code, people re-run the pipeline until it goes green, and eventually someone switches the test off. Studies put the time developers lose to flaky tests at 1.3–2.5% of their working time, and Atlassian reports over 150,000 developer hours a year in a single repository. GitLab's own engineering handbook runs a manual process for it: urgency tiers (48 hours, 1 week, 2 weeks) and a quarantine that expires after 3 months. FlakeProof automates that process with the tools GitLab ships.

AI tools that "fix" flaky tests already exist (Semaphore, JetBrains, BuildPulse). What bothered us is that they let the AI decide the fix worked, from a handful of green re-runs. A test that fails once in 1,000 runs passes 10 re-runs almost every time, so a few green runs prove nothing. We wanted the opposite: the agent proposes, statistics decide.

What it does

FlakeProof runs entirely inside one GitLab project:

  1. Hunt. On every push, a CI job runs the whole suspect test file 1,728 times in parallel under load and counts failures. If anything fails, the pipeline goes red.
  2. Diagnose and fix. The red pipeline triggers a GitLab Duo Agent Platform custom flow. The agent reads the job logs and the code, files an issue with a root-cause class and an urgency tier from the handbook policy, makes the smallest change to test code only, and opens a merge request. It is told explicitly that it may not claim the fix works.
  3. Prove. The merge request's flake-proof job reruns the same load 6,912 times. It merges only if all of these hold:
    • at least 300 runs and zero failures;
    • no other test in the file broke;
    • the drop against the baseline is statistically significant;
    • every changed file is test code.

It merges exactly the commit it proved.

  1. Ship and report. Main builds the container image, runs SAST, deploys to GitLab Pages, and posts the before/after numbers back on the issue. A nightly schedule re-hunts and audits expired quarantines.

The real run, end to end with no human step after the push:

runs failures
Before (pooled baseline, hosted runner) 6,048 7, all still loading
Hunt on the push that started the agent 1,728 1 → pipeline red
Proof on the agent's MR 6,912 0
Hunt on main after the merge 1,728 0

The agent found the cause on its own: line 96 of the shared render() test helper waits for the app to load with testing-library's default 1 s timeout, which a busy 2-vCPU runner sometimes exceeds. Its fix was one line, waitFor(..., { timeout: 10000 }).

How we built it

  • tools/flakehunt/: dependency-free Node scripts for the hunt runner, the statistics, the test-only gate, the verdict and merge, the issue report and the quarantine audit. 40 tests, written test-first.
  • .gitlab-ci.yml: tools, unit, flake-hunt, flake-proof, docker-build, SAST, pages and report, plus a calibration mode for choosing the stress level on the hosted runner.
  • .gitlab/duo/flows/flake-fixer.yml: a flow registry v1 custom flow with inline prompts, started by a pipeline-event trigger. It was created, enabled and given a pipeline-failed trigger through GitLab's GraphQL API (deploy/duo-setup.sh), with the trigger filtered to failed push pipelines on main.
  • .gitlab/flaky-policy.yml: GitLab handbook urgency tiers, the stop rule and the stress profile.
  • evidence/EVIDENCE.md: every number linked to the pipeline job that produced it.
  • deploy/: GitLab Pages for the live app, plus Google Cloud Run deployment code with keyless Workload Identity Federation (included; this entry is served from Pages).

All three sponsors in one agent run. GitLab's API reports the model behind our agent session as claude_sonnet_5_5_vertex: Anthropic's Claude Sonnet 5.5, served through Google Cloud Vertex AI, on the GitLab Duo Agent Platform. The design follows Anthropic's Building effective agents guidance that an agent should get "ground truth" from the environment at each step: here the environment, GitLab CI running thousands of tests, is also the only judge of the result.

The statistics. Zero failures in $n$ runs does not mean the failure rate is zero. With $n$ independent runs and no failures, the 95% upper confidence bound on the true failure rate $p$ comes from

$$ (1-p)^n = 0.05 \quad\Rightarrow\quad p \approx \frac{3}{n} \quad \text{("rule of three")} $$

so $0/6912$ bounds the rate below $3/6912 \approx 0.043\%$.

To show the change actually helped (rather than this being a lucky streak), the proof job compares against the baseline with a one-sided Fisher exact test. With $a$ failures in $r_1$ baseline runs and $c$ in $r_2$ proof runs, and $k = a + c$ failures in total:

$$ p = \sum_{x=0}^{c} \frac{\binom{r_2}{x}\binom{r_1}{k-x}}{\binom{r_1+r_2}{k}} $$

For $7/6048$ before and $0/6912$ after, $p = 0.0048 < 0.01$. The stop rule (≥ 300 runs, 0 failures, $p < 0.01$, no other test broken, test-only diff) was fixed before any fix was tried, and never changed afterwards.

Challenges we ran into

  • The flake is rare. On GitLab's hosted runner it hits about 0.1% of test runs. We had to calibrate a load that reproduces the same failure seen on ordinary runs (still loading). At 7–8 parallel processes we got a different, artificial failure (5 s test timeouts), so we froze the profile at 6. The low rate is also why the proof needs about 7,000 runs to reach $p < 0.01$.
  • The flake wasn't where the FIXME said. Calibration showed the FIXME test itself never failed (0/450). Failures spread over 7 different tests, all through the shared render() helper. We switched the hunt to the whole file.
  • Platform details we only learned by running it. Our first agent session died in setup (the flow's runner image has no yarn). The second diagnosed correctly but stopped before the fix (pipeline triggers carry no default-branch context). The merge API refused merges without the exact commit SHA. Each problem was fixed in code; both failed sessions are listed in the evidence.
  • CI gotchas. An unquoted : turned a YAML script line into a mapping. Node 20 does not expand quoted globs. vitest picked up our node:test files. GitLab omits jobs without rules from merge-request pipelines.

Accomplishments that we're proud of

  • A fully hands-off loop on a real project: push → red hunt → Duo agent → issue → fix MR → statistical proof → merge → scan → deploy → report, with no human step after the push.
  • The agent got it right on its own: it named the root cause, the file and the line, and listed what it could not verify. Its final message: "I haven't run any tests, and the flake-proof job on the MR is the only thing that can say whether it works."
  • Honest numbers: every figure links to a pipeline job, the stop rule was fixed in advance, and the failed attempts are documented rather than hidden.
  • Reproducible: one command sets up the Duo flow and trigger, and the CI calibration mode re-measures the stress profile on any runner.

What we learned

  • A green run means little for a rare flake. You need a number of runs chosen in advance and a pre-registered stop rule, or you end up fitting the test to the result.
  • The agent was most useful as a careful investigator that cites logs, files and lines and says what it doesn't know. Keeping it out of the merge decision made it easier to trust.
  • GitLab Duo flows, triggers and the AI Catalog are fully scriptable through GraphQL, so the whole setup can be reproduced with one command.

What's next for FlakeProof

  • Collect each failing test's name, so the agent sees exactly which test failed (it pointed out this gap itself).
  • Use coverage diffs of passing versus failing runs as extra evidence for the agent.
  • Hunt across the whole suite on a nightly schedule, with automatic quarantine markers and expiry.
  • Package FlakeProof as a GitLab CI/CD component plus an AI Catalog flow, so any project can adopt it in one step.

Built With

Share this project:

Updates

Submission history