Inspiration
When a test goes red in a large codebase, most of the time goes into finding the one function that broke, not into fixing it. I wanted an agent that does the whole loop a developer does: reproduce, localize, patch, run the tests, read what failed, and try again, and that stops only when the whole suite passes. I also wanted to know what it costs, call by call.
What it does
MutaFix takes a Python repository with a failing test suite.
- It runs pytest and collects the failing tests and their error lines.
- It re-runs the failing test files with a small tracing plugin (Python's
sys.monitoring) to see which library functions ran in each failing and passing test, and scores every function with the Ochiai spectrum formula. Ties are broken by static call distance from the failing test and BM25. - Nemotron 3 Nano (reasoning off) gets the first attempt. It reads code and proposes a patch through a JSON tool loop (
read,patch,give_up), about two seconds per turn. - Every patch is applied, then the failing tests and the full suite run. If anything is still red, the patch is reverted and Nemotron 3 Super (reasoning on) takes over the same conversation, sees which tests failed and why, and keeps going within a turn budget.
- Each run writes an HTML report: failing tests, trace, candidates with scores, every patch as a diff, every test result, the model's reasoning, and the tokens, latency and cost of each call.
The agent can only edit library functions, never tests; patches that do not parse never reach the disk; failing patches are reverted at once.
On a benchmark of 48 injected one-line bugs in toolz, more-itertools, boltons and sortedcontainers (pinned commits, answer hidden from the agent), MutaFix repaired 40 of 48 (the full test suite passes). 30 of those restore the original code exactly (same AST). Half of the repairs needed only Nano. The 48 tasks cost $0.39 in total, $0.0081 per task.
How we built it
Python, pytest and httpx, with every model call going to Nebius Token Factory's OpenAI-compatible endpoint. Three Token Factory features shaped the design: chat_template_kwargs.enable_thinking to switch reasoning per call, response_format: json_schema so every reply is a valid action, and the per-token prices published by /v1/models, which feed a cost ledger and a hard budget stop.
The design came out of measured experiments on the same 48 tasks:
- Static retrieval only (call graph + BM25): 31 repaired. The bug was missing from the 20 candidates in 11 tasks.
- Adding the test trace: 39 repaired; the bug was missing in only 1 task.
- A ranking-only experiment showed Nano re-ranking did not beat the spectrum order (top-5 hit 67% shuffled, 96% in spectrum order, same as the spectrum alone), so we stopped using Nano for ranking.
- Moving Nano to the first patch attempt kept 40 repairs and cut cost per task by about a fifth.
- Turning Super's reasoning off was cheaper still but repaired only 29.
All runs, traces and ledgers are in the repository's
results/folder.
Challenges we ran into
- Super sometimes spent its whole reasoning budget and returned nothing. We capped reasoning at 4096 tokens and ask again without thinking when that happens.
- The models re-read the same functions until they ran out of turns. A read budget, a visible turn counter and showing the top five functions in full fixed most of it.
- Models copied the line-number gutter into their patches or lost indentation. The patch matcher now tolerates both, but still rejects any edit that matches more than one place.
- Our first localization step was the bottleneck: 11 of 17 failures were bugs the agent never saw. The trace plugin fixed that.
- A 529 "overloaded" response stopped one benchmark run; the client now retries with backoff and the harness re-runs aborted tasks.
Accomplishments that we're proud of
- 40 of 48 bugs repaired, with every diff, test result and cent published.
- A measured reason for every design choice, including the one that removed an LLM step.
- A two-model cascade where the small model handles half the repairs at about two seconds a turn.
What we learned
A cheap, deterministic signal (which code ran in failing tests) beat an LLM at localization, and the LLM was more useful writing code. We also learned to report "tests pass" and "the original code is restored" separately: 10 repairs pass every test with a different edit, and in 2 of them the edit only hides the symptom.
What's next
Multi-function and multi-file patches, using passing-test coverage to propose new regression tests, Nemotron 3 Ultra as a third tier for the hardest cases, and running MutaFix in CI on real pull requests.
Log in or sign up for Devpost to join the conversation.