Flake Hunter

Inspiration

Every team I've worked with has that one test. Green on your machine, red on CI, green again if you rerun it and don't look at it too hard. We had one that ate three days of a release cycle before someone just deleted it. That's the dirty secret of flaky tests: nobody fixes them, they delete them or mute them, because finding why a test fails 1 run in 20 means running it 100 times and diffing traces by hand.

What it does

Flake Hunter runs your test suite 50 times per test across a parallel fleet of sandboxes, proves statistically which tests are flaky (with confidence intervals, not vibes), root-causes each one, and opens a pull request with the fix - plus the measured before/after flake rate so you can see the fix actually worked.

How I built it

Test runs fan out across Nebius Token Factory Sandboxes orchestrated by Nebius Serverless Jobs. Failure traces are triaged cheap with Nemotron-3-Nano, and confirmed flaky tests go to Nemotron-3-Ultra for deep root-cause analysis and patch generation. The patch gets verified the same way the flake was caught - 50 more runs - before a GitHub App opens the PR.

Challenges

Making 100 sandboxes feel instant in a dashboard, and getting an LLM to write patches that survive statistical verification instead of just sounding right.

What I learned

The difference between a flaky-test detector (everyone has one) and a fix anyone trusts is measurement. 48% -> 0% beats "the AI says it's fixed" every time.

Accomplishments that we're proud of

A patch isn't done when the LLM writes it - it's done when 50 fresh runs say the flake is dead. Getting that verify loop airtight (48% flake rate to 0%, measured, in the PR body) is the thing nobody else in this space does.

What's next for Flake Hunter

Monorepo-aware sharding, flaky-test trend history so teams can watch their suite get healthier over time, and a one-click GitHub Marketplace app so any repo gets its first audit in minutes.