Open the live demo · Code · A pull request it opened from a red CI build
Inspiration
Most coding agents available today share one shape: edit, run, read the error, edit again. Watching them work, three costs keep showing up — and none of them are about the model.
When attempt #3 makes things worse, the agent has to undo, and until it finishes undoing it is reasoning on top of a repository it has already damaged. Most agents do not even try; they carry the damage forward. Second, the expensive part — clone, install, build, warm the cache — gets repeated for every candidate patch, because each attempt starts from a fresh container. Third, and worst: a bug usually has two or three plausible explanations, and a linear agent picks one, commits to it, and never learns that the second was right.
Then Nebius Sandboxes turned out to have Git-like branching over container execution state. Every command produces a new immutable filesystem version, and you can fork from any of them. All three costs stop being costs.
What it does
Arborist takes a repository whose tests fail and returns a patch that makes them pass — by searching a tree of repository states rather than walking a line of edits. It is built on Nebius Token Factory: NVIDIA Nemotron 3 for every model call, and Nebius Sandboxes for every execution.
It is for engineering teams whose CI goes red: instead of a developer stopping to bisect a failure, the build answers itself with a review-ready pull request.
It runs the setup once and keeps that checkpoint. NVIDIA Nemotron 3 Super reads the failure and produces several genuinely different theories of it. Each theory becomes a patch written by NVIDIA Nemotron 3 Nano, applied to its own fork of the same checkpoint, and tested independently. The tests score every fork; the best one becomes the next fork point; anything that regresses is abandoned at zero cost, because nothing was ever mutated.
What comes out is a patch and the record of what was rejected, opened as a pull request. Every branch the search explored is in the body — folded away, but present — with the test-level reason it lost, so a reviewer can tell a right theory implemented badly from a wrong theory. Most coding agents hand over only the final diff; this is the difference between trusting a diff and being able to check one.
It works three ways:
- From the terminal:
arborist fix .on any Python repository, with the tests locked against edits (--protect) and the issue text as the goal. - In CI: a GitHub Action. Add one step after the tests, and a red build
answers itself with a pull request — opened by
github-actions[bot], with the search in the job summary and a spending cap per run. - In the browser: every search replays as a tree you can click through — each branch's theory, its diff, the tests it fixed and the ones it broke — on desktop or phone.
How I built it
Three NVIDIA Nemotron 3 tiers on Nebius Token Factory, routed by what each step
is worth. All three are called through Token Factory's OpenAI-compatible
endpoint with json_schema output — one base URL, three model sizes, so moving a
step between tiers is a one-line config change.
- Nemotron 3 Nano (30B A3B) writes every candidate patch — one call per branch, the widest step in the system. Most branches get thrown away, so the discarded work has to be the cheap work.
- Nemotron 3 Super (120B A12B) diagnoses each node and produces the hypotheses the search branches on. This is the reasoning that shapes the tree: two hypotheses that would produce the same edit are two wasted branches.
- Nemotron 3 Ultra (550B A55B) is woken when the search stalls — two expansions without improvement — or when two branches score identically. That second case is the interesting one — when the tests cannot separate two patches, the suite has stopped being informative, which is exactly when one of them is gaming it. Ultra is asked specifically to catch a patch that passes by weakening an assertion, swallowing an exception, or special-casing the test input, and to flag it rather than let the score stand.
Nebius Token Factory Sandboxes (ConTree) is every execution. base() builds the expensive
prefix once, run() forks from any checkpoint, read() pulls the JUnit report
back out of one specific state. Branching is not an optimisation here — it is the
mechanism the whole design rests on.
Tavily is consulted whenever the diagnosis implicates a third-party package, asks for documentation, or reports low confidence in itself. The first of those was added after a measurement. Asking only when the model volunteers "I don't know" turned out to be too narrow: on a case where a Pydantic 1.x model sits in a project pinned to 2.x, Nemotron was confident, never asked, and never searched. A model is confidently wrong about a library exactly when its training snapshot predates the version in the repository — and it cannot know it is in that case, so its own confidence is not a usable filter. One lookup costs less than one wrong patch and the sandbox execution behind it. A confident, repo-local diagnosis still never touches the network.
Two design decisions did most of the work:
- Scoring uses JUnit XML, not stdout scraping, so "fixed" and "broke" are lists of named tests compared against the parent state. That is what lets the search see a trade — fixed two, broke one — and refuse to follow it. Pass rate alone would have called that progress.
- A patch that does not apply is rejected before execution. Every edit quotes an exact snippet that must appear exactly once in the file. An invalid patch costs tokens, never wall-clock.
Challenges I ran into
Building a comparison nobody could argue with. A linear baseline is easy to handicap by accident, so I saved every run's full tree and read them. That turned up five places where the harness quietly favoured branching — a shared depth cap, early stops, estimated setup counts, a failed patch dropping its parent state, and an uneven per-state attempt budget — and a patch-application bug losing three branches in five. All fixed before the final measurement; none would have shown in a results table.
Keeping hypotheses genuinely different. Branching only pays if the siblings are rival theories, not rephrasings of one. The diagnosis prompt demands distinct hypotheses and every expansion is told what its branch already tried, so each fork explores something new.
Scoring that sees a trade. It took switching to per-test identities, compared against the parent state, before the search stopped chasing trades that looked like progress.
An SDK ahead of its docs. The published Sandboxes Python guides describe
Contree(api_client) taking a pre-built client, but the current stable release
on PyPI exposes ContreeSync(token=…, base_url=…) — the client-injection API is
only on the 0.4.0 dev releases. I found this by introspecting the installed
package rather than trusting the docs, and pinned the released version.
Accomplishments that I'm proud of
A search that wins where it matters, measured on real infrastructure.
On Nebius Sandboxes, same models, same prompts, same budget, the tree search solved 11 of 12 runs across the four test cases against a linear agent's 6, ran the environment setup once per run instead of seven times, and spent 60% less on models. On the two cases built to need exploring it was 5 of 6 against 0 of 6.
masked-faults shows why. An import error stops the suite collecting, so two
further faults give no signal until it is repaired — and then the obvious fix,
the one the failing test is named after, moves nothing, because a second fault
still hides the rows that would prove it. The search put three rival repairs on
one checkpoint, saw the obvious one come back neutral while its sibling reached
3 of 4, and forked from the sibling to green. The linear agent spent ten of its
twelve patches on one state, unable to tell a wrong theory from a wrong
implementation of a right one — because it never had a sibling to compare with.
Real repositories, real savings. On 23 SWE-bench Lite issues from flask, seaborn and pytest, run inside the benchmark's own images on Sandboxes, Arborist resolved 14 — each re-verified from a clean checkout — with half the patches, 43% less time and a third of the model spend of a linear agent. A linear agent that stalls keeps escalating to Ultra; a search that keeps finding progress rarely needs it: 29 Ultra calls across 35 branching runs, against 196.
Evidence anyone can click. Every number is checked in and stamped with the commit that produced it, the run trees behind the claims replay in the public demo, and one of them became a real pull request that lists the alternatives the search rejected and the tests each one broke.
Engineering that made the search pay off. Patches are validated before they run: a failed one gets a single repair attempt with the reason fed back, which took unusable patches from 3-of-5 to 0-of-8. Whole-file rewrites are stripped of formatting-only changes, so a diff holds the fix and nothing else. The tests can be locked, so green means the code was fixed, not the oracle. And the whole search runs offline too — a scripted model stands in for Nemotron and a directory-snapshot backend for Sandboxes — so 182 tests exercise selection, scoring, validation, backtracking, API and CLI with no network and no keys.
What I learned
Two things.
That the interesting constraint in agent design is often infrastructural, not model-shaped. Nothing in this project needs a smarter model. It needs execution state you can fork — and once you have that, a whole class of agent behaviour that was previously unaffordable becomes ordinary.
And that an agent's bottleneck is rarely where you think it is. Search strategy mattered — but so did whether a patch could be applied at all, and whether the baseline was given a fair chance. Both only became visible once I saved the search tree and looked at it, which is the same argument this project makes about pull requests: the diff is the answer, and the discarded branches are the reason.
What's next for Arborist
A GitHub App, so every repository in an organisation gets the red-build repair without adding a workflow step. Language runners beyond pytest. And carrying a scored branch tree across runs on the same repository, so the agent starts a second bug already knowing which theories that codebase tends to reward.
Log in or sign up for Devpost to join the conversation.