Flaky Tests Detective

Inspiration

Every team has that one test. Green on your machine, green on the rerun, red exactly when you're trying to ship. Flaky tests are a special kind of misery because the CI log tells you that something failed, never why so debugging them means rerunning, guessing, and sprinkling sleep() until the red goes away. That "disciplined guessing" loop is mechanical and repetitive, which made me wonder: could you automate the whole thing, from a red CI run to a reviewed, green pull request?

What it does

Flaky Test Detective runs a failing pytest test through a five-stage pipeline: ingest → reproduce → classify → fix → PR. It reproduces the flake deterministically, identifies which classic flake class it belongs to (race condition, time dependency, shared state, order dependency), writes a minimal cause-specific patch, and only if that patch survives re-testing opens a pull request with a plain-language explanation and a before/after evidence table. I demonstrate the full red→green arc across all three core flake classes using a suite of planted fixtures with known ground truth.

How we built it

The heart of the project is one idea: perturbation is both the forcing function and the diagnosis.

Naively rerunning a flaky test is a bad strategy. If a test fails with per-run probability $p$, then running it $N$ times and seeing it pass every time happens with probability $(1-p)^N$. For a genuinely rare flake say $p = 0.02$, even $N = 20$ reruns miss it two-thirds of the time: $(0.98)^{20} \approx 0.67$. You can rerun all day and wrongly conclude everything's fine.

So instead of hoping, the detective forces each flake class with a targeted perturbation and records a failure-rate matrix:

The shape of that matrix is the classification signal a test that only fails under jitter is a race; one that only fails under a frozen clock is time-dependent. An OpenAI model (via the Responses API) reads the matrix, the failure output, and the test source to produce a structured verdict but it's never trusted blind. The response is forced through a strict JSON schema and re-validated in code before it can influence anything downstream.

Challenges we ran into

Reproduction was the whole battle. Getting a flake to fail on demand reliably, in an isolated subprocess, without ever touching the original files took far more iteration than the fix templates did. The jitter and clock-freeze perturbations are injected through a generated sitecustomize.py shim, and getting that to load in the subprocess's import path was genuinely fiddly.

Keeping the LLM on a leash. An LLM that's the sole judge is a single point of failure it can be confidently wrong. Threading it between deterministic signals and a hard validation gate, so its verdict is always checked against reproducible evidence, was the core design tension of the project.

Statistical honesty about "fixed." Zero-tolerance validation is reassuring but not free. If a patch leaves a residual failure rate $q$, it still passes an $N$-run check with probability $(1-q)^N$ — at $q = 0.01$, $N = 100$, that's $(0.99)^{100} \approx 0.37$. Perturbation is what makes acceptance meaningful: it drives the residual rate under the forcing condition up to where $N$ runs can actually catch it. Making the gate statistically rigorous (a threshold instead of a fixed $N$) is on the roadmap.

Accomplishments that we're proud of

The part I'm proudest of is fix validation, which I think of as "prove it with the same weapon that killed it." A patch is accepted only if rerunning the original perturbation matrix on a patched copy drops the failure rate to zero across the board. Nothing reaches a PR on the strength of a plausible-looking guess.

What we learned

• Each classic flake class has a distinct forcing condition and you can weaponize that for both reproduction and diagnosis at once.
• How to make an LLM a useful component of a pipeline rather than the pipeline itself, bounded by a schema on one side and empirical validation on the other.
• A lot about Python's import machinery, subprocess isolation, and generating and applying unified diffs programmatically.

What's next for Flaky Tests Detective

• Generalize the fix strategies beyond the fixture patterns into AST-based transforms that handle real-world variation in how these bugs are actually written.
• Harden the repo → PR path for arbitrary repositories: authenticated pushes, correct workflow permissions, and default-branch detection so it runs cleanly inside GitHub Actions.
• Graceful degradation — open a findings issue when a flake can't be reproduced or confidently fixed, instead of stopping.
• Statistical validation thresholds and support for more flake classes.

Built With

• codex
• python
• uv