Patchwright

An AI bug-fixer on Nemotron and Nebius Token Factory that ships only fixes plain code has proved, and uses the expensive model only where needed. Real PR, honest numbers.

Comment

Inspiration

AI coding tools write a plausible fix and leave you to check it. A wrong fix that looks right costs more than no fix. We wanted to move the trust from the model to code: the model proposes, plain code judges. And because the biggest model is expensive, we wanted to measure exactly where it is worth paying for.

What it does

Patchwright takes a bug report and a repository, reproduces the bug in a Docker sandbox, writes a failing test, fixes the code, and only reports success when an independent harness confirms all of this:

  1. the new test fails with the fix reverted,
  2. the same test passes with the fix restored,
  3. the project's existing suite still passes and no existing test was edited,
  4. a reviewer's edge-case tests find nothing.

It delivers a clean patch (scratch files left out) and a pull-request description with the proof, tokens, cost and sources. It never merges anything.

Three NVIDIA Nemotron 3 models on Nebius Token Factory share the work: Nano (30B) triages and judges whether the task is a real, testable bug; Super (120B) writes the failing test and the minimal edit; Ultra (550B) takes a fresh attempt on a reset repository only when Super is stuck.

A real bug in a real library

We ran it on a real, closed upstream issue (tkem/cachetools#423) using only the issue text, and judged the result with the upstream project's own test, which the agent never saw. Result: a real pull request with a 2-line fix and a test, written by Nemotron 3 Super in 14 steps for about $0.055. It took seven runs; two solved it. Each of the five failures exposed a real flaw in Patchwright (a destructive whole-file rewrite, a garbled edit that deleted a method, scratch files in the patch, wandering through huge files). We fixed each with a regression test and published all seven runs.

What we measured (real Nebius billing prices)

On 20 bugs with hidden tests, 40 runs per strategy, compared on the same runs:

  • Super alone fixes 50% of the bugs, Ultra alone 82.5%, tiered (Super then Ultra) 85%. Tiered matches Ultra's fix rate at about 26% lower cost per fixed bug ($0.059 vs $0.080), because Ultra costs about 3.3x Super per token.
  • The biggest lever was not the router, it was fewer model calls: the whole conversation is re-sent on every call, so about 75% of tokens are prompt tokens. Letting the model send up to three actions per reply cut cost per fixed bug by 51% (tiered, $0.048 to $0.024) and 67% (Ultra-only, $0.087 to $0.029) on the same 20 bugs, with no loss in fix rate (19 of 20 each).
  • A negative result, reported honestly: asking Nano to rate difficulty did not predict which bugs a larger model would fail, so difficulty routing added nothing. Nano was reliable at refusing non-bug requests (10 of 10 in a small check) and wrongly refused 1 of 5 real bugs, so the app offers "Run anyway".

How we built it

Python, FastAPI and a no-build single-page front end; a Docker sandbox per run; an OpenAI-compatible client for Token Factory; a harness that reverts and restores the fix itself; a 20-bug benchmark with hidden tests and reference-fix validation; spend limits and loop protections tested with a scripted fake model so they cost nothing; 132 tests and CI.

Tavily is used by the harness itself, not just offered to the model: when a command fails with a library or version style error it searches for it, and for a real repository it looks up related discussion and official docs before the model starts. Every query and source is shown in the live view and in the pull request. Measured: on the synthetic benchmark the docs lookup cost 26% more tokens for no gain, so it is real-repo only; on cachetools it found related issues and docs, not the fix.

Challenges

Our first real-world batch fixed 0 of 7, mostly our own fault: Nemotron 3 reasons by default and used the whole reply budget in an agent loop (turned off: 37 tokens and 1 s instead of 3,000 tokens and 28 s with no answer); models invented action names, so we moved to native tool calling; they read whole files instead of editing, so we improved code localization; old test suites already failed, so a fix is judged on the failures it adds.

What we learned

Measure the cost of a fix, not a call; publish the failures.

Honest limits

The 20-bug benchmark is synthetic: the final code fixes 18/20 for about $0.01 per fixed bug. On real issues (Patchwright-RW: boltons, tabulate, toolz, judged by upstream tests) 3 of 14 runs were fixed, and the cachetools PR task 2 of 7. Small samples; Python projects with tests only.

What's next

More languages, a larger real-world set, PRs from the UI. Built from 2026-10-20. Code: https://github.com/rohith95-p/patchwright (MIT).

Built With

Share this project:

Updates

Submission history