Login
Overview
Scan repository lists
Import repo from public url
Github authorized import private repo
Repository Runs
Repository Runs Failure
Flakes - Patch
Flakes - Patch
Flakes - Evidence
Flakes - Evidence
Flakes - References
Flakes - Escalations
Integrations
Settings Defaults

About the Project

A test that fails on every run is broken. There is no point of chasing. A test that fails sometimes is a flake.

Deflake is that missing piece. It finds flaky pytest tests, names the cause, writes a small fix that has to change the cause, not to hide it and proves that fix with repeated runs, and routes a strong proof toward a pull request, or a weak one to a person with the evidences.

Built on Nebius Token Factory with NVIDIA Nemotron models (Nano, Super, and Ultra), orchestrated with LangGraph.

It reruns the suite many times in random order, with a fresh hash seed each time.

For each flake it names a category:

order,
a race,
the clock,
randomness,
a shared resource,
hash order,
or the environment.

Backstory in short

It is Thursday evening and the pipeline is red. The same commit that successfully passed an hour ago is now at red cross. The job was re-run, it went green and the release ships.

By next morning, teams are wrapped with multiple retries. Some went through and sometimes it fails. Looks like real bugs slip through because the test is always "flaky".

People were using cursor and the coding agents make it worse. They hide the flake with a sleep. The suite eventually looks green, but the race of running into red still exists.

I always wanted to put this same monotonous issues into a kind of automated solution. I don't know where should I start and what right set of tools required. Going through articles, reading about problems, understanding the issues involved and collecting softwares. But it was a kind of pieces, scattered all over the internet. To solve, it requires a set of step-by-step procedures (diagnosis, test cases, prove the patch, confirm fixes before apply, etc) to be carried out initially by engineers before jumping into actual fixes. Feeling clueless how to put these things together?

How it works

Use public repos link, or connect and authorize (OAuth) to Github and start scanning your private repos.
Every model call goes to Nebius Token Factory through LangChain, orchestrated with LangGraph.
For each flake, Nano classifies the failure and drafts a search query. Tavily looks for matching bug reports.
Ultra diagnoses the cause three times. We only keep the cause when the samples agree. If order matters, we bisect until we can name the polluter.
A 1 out of 3 split does not get papered over, it goes to a person.
Super writes a minimal patch. Guardrails reject sleeps, retries, skips, deleted tests, and weakened assertions and that rejection stays visible.
Strong proof above the confidence bar can be marked for an automatic pull request.
Weak score, failed proof, or no safe patch — Tendem gets the runs, the diagnosis, and every patch attempt.
Toloka reviews whether a cause actually matches its patch, so the confidence bar isn’t just a guess.
LangSmith for observability on LLM responses

While testing on a repo that has a project ecommerce-shop, five fixes came back proven, namely coupons, inventory, orders, pricing, revenue. The speed check and the environment case were escalated instead of faked.

A green test is not proof. Deflake doesn’t stop at detection. It diagnoses, fixes, proves, and knows when to ask a human.

That’s Deflake - flaky tests, real causes, proven patches.