Inspiration
Git bisect assumes the test tells the truth. On a flaky test it doesn't, and bisect doesn't notice — it commits to a branch on one observation and never revisits it. The answer it gives is confident and wrong, and it's a different wrong answer each run.
What it does
Four stages. Each one is verified rather than asserted.
0. Which test. Given a bug report in prose, it writes candidate test scripts and keeps only the ones that measurably fail more often at the bad revision than at the good one. A candidate that always passes, or that is pointed backwards, reads as perfectly reasonable source code — it gets rejected by running it at the two known ends, not by reading it.
1. Which commit. k-ary bisect where every probe is a Wald sequential probability ratio test against two measured failure rates. A revision is re-run until the evidence crosses a boundary, then it stops. With --fast, probes that have stopped mattering are cancelled and the next round starts on a prediction of the slow one.
2. Which lines. Delta debugging over the culprit commit's diff hunks, down to a 1-minimal set — with a real third outcome for "the patch doesn't apply", which is unresolved, not rejected.
3. Which fix. k candidate patches from a model, each applied to the culprit and put through the same sequential test at the same alpha. Only the survivors are reported.
How we built it
Python, standard library only. Nebius Token Factory for inference (nvidia/Llama-3_1-Nemotron-Ultra-253B-v1), and ConTree sandboxes for probe execution: the repo is cloned once into an image, each revision is one checkout kept as a checkpoint, and every test run forks from that checkpoint disposably. How many runs a probe needs isn't known until it stops needing them — which is the case for disposable microVMs rather than a fixed pool of workers.
Challenges we ran into
Model-written unified diffs get line numbers wrong. So the model quotes the exact text it wants replaced and difflib computes the diff. A SEARCH string that matches zero places, or two, drops the candidate instead of guessing which one was meant.
And a label is not evidence. Handing stage 3 only "testing.py:116" plus the file gave zero usable patches. Handing it the minimized diff — the thing stage 2 actually proved — gave four out of four.
Accomplishments that we're proud of
On pallets/click #2939, over 234 revisions: the right commit at index 231 in 16 probes, one hunk out of eight, and two verified fixes, one of which is what the maintainers actually shipped.
Three identical runs produce bit-identical stages 1 and 2. Only the model's output moves, and the verification gate makes that harmless.
What we learned
Variance belongs to the model, not to the search. You don't remove it, you gate it.
What's next for bisect-agent
A wall-clock comparison of local worktrees against ConTree forks on a repository big enough for checkout cost to dominate, and grouping hunks that have to move together, so stage 2 can also minimize a commit that refactors and breaks in the same change.
Try it
git clone https://github.com/gigaminkyu/bisect-agent
cd bisect-agent && python -m bisect_agent --selfcheck
Built With
- contree
- delta-debugging
- nebius-token-factory
- nvidia-nemotron
- python
- sprt