FlakeFix

A self-verifying agent for flaky tests: detects, diagnoses, fixes, and reruns until proven — then opens the PR. It never says fixed until it's watched the fix pass twice.

Comment

Inspiration

Flaky tests are one of the most universally hated problems in software engineering — a test that sometimes passes, sometimes fails, for no code change at all. The real-world response is almost never "fix it": it's @pytest.mark.flaky, a retry decorator, or just re-running CI until it goes green. The bug stays in the codebase forever, quietly eroding trust in the whole suite.

We didn't want to build another AI code assistant that writes a plausible-looking fix. Every tool does that, and you can never tell if the fix is real or just confident-sounding. We wanted to build something that could prove its work — that literally cannot claim success unless it has watched the fix pass, twice, in isolation.

What it does

FlakeFix is a closed-loop agent for exactly one bug class: flaky pytest tests. Point it at a repo — a local path, or just owner/repo and it forks and clones it for you — and it will:

  1. Detect — run the suite repeatedly with shuffled ordering, flag tests with mixed pass/fail histories.
  2. Reproduce — isolate the flake, identify order-dependent polluting tests, capture hard evidence (diffs, tracebacks, pass rate).
  3. Diagnose — classify the root cause into one of five categories (unseeded randomness, live clock, shared mutable state, fixed sleep timing, ambient filesystem state), with cited evidence lines. OpenAI-backed when configured, deterministic static rules as an always-on safety fallback.
  4. Fix — generate a minimal, category-specific patch.
  5. Verify — apply the patch in a disposable git worktree (your working tree is never touched), re-run the fixed test to a required 100% pass rate — combined with any identified polluting test, not just alone — then re-run the entire suite and reject any regression.
  6. Deliver — commit the verified fix locally, and, opt-in, push it and open a PR via gh with the evidence and reasoning in the PR body.

If any step can't be proven, FlakeFix says so — FAIL_STILL_FLAKY, FAIL_REGRESSION, COULD_NOT_VERIFY — and never reports a fix as done unless it watched it pass.

How we built it

Detection shuffles the full test-id list with a deterministic per-run seed and runs the suite 8-15 times, parsing structured JUnit XML per run. Reproduction re-runs the flaky test in isolation; if it only fails combined with other tests, we probe up to 8 preceding tests to find the actual polluter. Diagnosis tries an OpenAI model first when configured, but its answer is only accepted if it names a category we actually have a patch generator for, cites real line numbers, and gives real reasoning — otherwise it falls straight through to static rules. Verification never touches the caller's tree: a disposable git worktree on its own branch, a required 100% reverify pass rate, a full-suite regression check, and only then a local commit. A live control-room dashboard streams every stage in real time, and flakefix fix <repo> collapses the entire loop — fork, clone, detect, diagnose, fix, verify, PR — into one command.

Challenges we ran into

The honest ones, because that's the whole point of this project:

  • Two of our five original "fixes" were quietly gaming the test's assertion instead of fixing root cause (freezing a clock to a value that trivially satisfied the check; deleting a file right before asserting it didn't exist). We caught this in an internal audit and rewrote both as real fixes — frozen clock is a legitimate technique (same as freezegun), but only once the assertion still means something.
  • A demo test had a repo-litter bug: it wrote a marker file and never cleaned it up, so after the first bad run it failed permanently, not flakily — and polluted the real repo on every future run. We rebuilt it as a genuine two-test ambient-state leak, the actual real-world shape of that bug class.
  • Our git worktree verifier reverified order-dependent fixes in isolation only — meaning a "100% pass rate" for a shared-state fix never actually re-exercised the interaction it claimed to fix. We rebuilt reverification to run the target test together with its identified polluter.
  • Ran it for real against Delgan/loguru (24k stars) — forked, cloned, and tested via gh, live. It found 0 flaky tests across 3 real timing-dependent files. Not a bug: an honest negative. That's the whole pitch working exactly as designed — it didn't fabricate a flake to give us a demo moment.
  • A branch-naming bug: picking the next verification branch name by counting existing branches broke the moment that sequence had a gap, causing a real collision and a false COULD_NOT_VERIFY. Fixed by checking each candidate name's existence directly instead of counting.

Accomplishments we're proud of

A tool whose safety guarantees are load-bearing, not marketing: local-only by default, PR-opening is opt-in and pinned to your own fork so it can never open a PR against a repo you don't own, and every verdict is one of four honest words, never a guess. 5/5 supported flake categories verified end to end, 45.1 seconds, 26/26 internal tests passing.

What we learned

The hard part of "AI fixes your bug" was never writing the patch — it was proving the patch is real. Every shortcut we caught ourselves taking (gaming an assertion, reverifying in isolation, counting instead of checking) looked fine until we actually ran it against something real.

What's next

Support for class-based test methods (currently module-level functions only), async/thread race repair, and running the closed loop against more real-world repos to find and fix a genuine flake live, not just prove the honesty guarantee on a negative.

Built With

Share this project:

Updates