Inspiration
It started with a bug that didn't look like one.
In April 2026, TorchIO, one of the most widely used medical-imaging libraries in Python, shipped a fix for a quiet problem. When it handed an image to SimpleITK, it passed along an affine in RAS coordinates without converting it to LPS, the convention SimpleITK expects. Every image saved that way moved, by up to 123 mm. Nothing crashed. No warning appeared. The file opened fine in a viewer and the anatomy looked normal. It was just in the wrong place.
That idea stuck with us. A medical image is more than pixels: it carries a map that says where every voxel sits in the patient's body, in millimetres. Code that crops, flips, resamples or reorients a scan has to keep that map honest, and when it doesn't, nothing tells you. Brains are nearly left-right symmetric, so a flipped scan looks perfectly reasonable to a human. Researchers have found wrong orientation in the headers of public MRI datasets to be "surprisingly common" for exactly that reason (Glen et al., 2020).
Then we thought about AI coding agents, which are increasingly the ones writing and fixing this kind of code. If a person can't see the bug, and the unit tests don't catch it, what is the agent supposed to check its fix against? We tried the obvious thing: we told a strong model that a script was "reported as wrong" and asked it to fix it. When the script was actually correct, the model rewrote it anyway in 16 to 26 of 33 runs. When nobody had ever declared which convention the code should follow, it picked one in 9 to 16 of 21 cases, even with an explicit "abstain" tool available.
That isn't really the model's fault. It did what the request implied. The real gap is the missing check, one that can say "this is already correct" and "nobody can know the answer here". So we set out to build that check, and then to give it to NVIDIA Nemotron and see how far it could go.
## What it does
SpatialContract finds the bugs that put medical images in the wrong place, repairs them with NVIDIA Nemotron, and verifies every fix against physical ground truth.
- You declare what the code promises. A pipeline states a four-field spatial contract: which world frame (RAS or LPS), which units, which affine convention and, optionally, which array orientation.
- An oracle measures where things land. It pushes a synthetic phantom with landmarks at known physical positions through the code and reports, in millimetres, where each landmark ended up. It also describes the mistake in words: "x and y negated", "shifted by a whole number of voxels", "scaled by 1000".
- Nemotron repairs the code. Nemotron 3 Super, running on Nebius Token Factory, reads the code and the measurements and proposes a patch as a tool call. The oracle re-checks it, and the loop repeats until every landmark is back on target.
- Every fix faces checks the model never saw: hidden phantoms, a phantom generated after the patch was written, the same image stored in a different orientation, and an ordinary image checked with SimpleITK.
- It refuses to guess. If the contract is missing or ambiguous, the system abstains instead of picking a convention.
- It works inside real libraries. A runtime trace ranks the library functions most likely to hold the bug, so the agent starts from a short list instead of searching blind.
- You can use it without the agent. There's a Python decorator, a pytest fixture, a
spatialcontract checkcommand with CI exit codes, and a GitHub Actions workflow.
The demo page replays 12 real recorded runs in 3D, including the failures. It also shows a real brain scan before and after Nemotron's fix, and walks you through trying it yourself.
## How we built it
The contract. Every pipeline declares a small dictionary at the top of the file:
SPATIAL_CONTRACT = {
"world_frame": "RAS", # or "LPS"
"units": "mm",
"affine_convention": "voxel_center_to_world",
"array_orientation": "RAS", # optional
}
We read it with ast.literal_eval, so nothing is ever executed. Missing or "unknown" fields make everything
downstream abstain.
The oracle is deterministic, with no LLM in it. It builds a phantom: a 3D volume holding six distinct balls at known positions. The volume is stored in a random non-RAS orientation, with anisotropic voxels and sometimes an oblique affine, which are exactly the conditions that expose orientation bugs. It runs the code in a sandboxed subprocess, finds each ball in the output, maps it through the output's own affine, and measures the error:
$$ e_i \;=\; \bigl\lVert\, A_{\text{out}}\,\tilde{v}_i \;-\; p_i \,\bigr\rVert_2 \quad\text{(mm)}, $$
where $\tilde{v}i$ is the homogeneous voxel coordinate of landmark $i$ in the output, $A{\text{out}}$ is the output affine, and $p_i$ is the true position in the contract's frame. Lossless pipelines must land within $0.05$ mm. Resampling pipelines get a tolerance that scales with voxel size, capped at $4$ mm.
Six landmarks are enough to recover the transformation the buggy code actually applied, so the oracle fits it by least squares,
$$ \min_{M,\,t}\; \sum_{i=1}^{6} \bigl\lVert M p_i + t - q_i \bigr\rVert_2^2 , $$
where $q_i$ is where landmark $i$ actually landed. It then describes $(M, t)$ in plain words. That's a measurement, not a hint: it never names a line or a fix.
The repair agent uses nvidia/nemotron-3-super-120b-a12b through Nebius Token Factory's OpenAI-compatible API
with tool calling. For scripts, Nemotron returns the whole patched file through submit_patch. A guard rejects unsafe
code and any change to the contract, so the model can't redefine what "correct" means. We tried Nemotron 3 Nano
first because it's cheaper. It repaired 2 of 3 trial bugs, and fewer with reasoning turned down, while Super repaired
all 3 and used fewer tokens per solved case.
Inside libraries, we reproduced eight real, already-fixed bugs from TorchIO, nibabel, MONAI and nilearn at their pre-fix commits. Before Nemotron sees anything, we run the code under a profiler on phantoms it gets right and wrong, and rank every executed library function with an Ochiai score:
$$ \text{Ochiai}(f) \;=\; \frac{n_{\text{fail}}(f)}{\sqrt{N_{\text{fail}}\,\bigl(n_{\text{fail}}(f) + n_{\text{pass}}(f)\bigr)}} , $$
plus a bonus for functions that returned the exact affine written to the output. Nemotron then edits a private copy of
the library with read_file, search, replace_lines, run_oracle and get_suspects. We also built a staged
mode: Nemotron rewrites the top suspect functions three ways per round, and the oracle picks the best.
Verification beyond phantoms. Library repairs must also pass the library's own tests and keep its loader reporting the same affine. Every repair was then run on four real scans, including the Colin27 brain stored in two layouts, where the input scan is the ground truth.
Everything is Python: NumPy, nibabel, SciPy, SimpleITK and pytest, with 150 tests. A persistent token ledger and a dollar cap guard every API call.
## Challenges we ran into
Honestly, making the evaluation trustworthy was harder than building the agent.
Our own tools lied to us more than once. In the first full library run, the agent's search tool silently returned
"no matches" for perfectly normal glob patterns, so the model was searching blind. We only caught it by reading the
logs line by line, and we threw the whole run away. Mid-evaluation, we noticed that the model could read a comment in
one of our bug files that described the bug. We deleted every such comment and discarded that run too.
The oracle needed the same scrutiny we gave the model. It missed a crop bug because the phantom's bounding-box margins happened to be palindromic, so a wrong crop looked correct. It false-alarmed on a coarse resampler. It missed a one-voxel nilearn bug that sat under its tolerance, and it crashed on a degenerate affine. Late in the project, an outside review found that correct pipelines using linear interpolation or intensity normalisation were failing the oracle, because landmarks were located by exact label value. We fixed each one, re-ran everything, and confirmed that no earlier verdict changed.
The most humbling moment came from the creative repair methods. A model-free mutation search "fixed" TorchIO's crop bug by deleting the crop entirely. Two staged repairs made TorchIO's reader 134 mm wrong, while keeping the round trip through our driver correct. All of them passed every phantom check, because the oracle only sees what the driver exercises. So we added gates that come from the library itself, its own tests and its loader. They rejected three repairs that would otherwise have counted as wins.
Budget mattered too. The promo credits were finite, so we built hard token and dollar caps, and a client that stops for good on the first out-of-credits error.
## Accomplishments that we're proud of
- It catches what tests miss. The oracle detects 8 of 8 real bugs that shipped in public libraries. At each pre-fix commit, the libraries' own covering tests passed: 0 of 8 caught.
- Nemotron repairs well with the right check. With oracle feedback it repaired 98% of our 31 seeded bugs, modified 0 of 33 correct scripts, and guessed a convention in 0 of 21 cases where none could be known.
- The fixes hold on real anatomy. All 91 phantom-validated repairs pass on four real scans. All 16 library repairs produce output identical to the upstream fix on every scan (64 of 64). On the demo, a real brain's buggy output correlates 0.68 with the input, and Nemotron's repair 1.00.
- It finds the bug without a model. The runtime trace puts the file the real fix edited in its top 10 for 8 of 8 bugs. On TorchIO's crop bug, 101 functions ran and the right one ranked sixth.
- Real library fixes, cheaply. Across our methods, Nemotron produced verified fixes for 4 of the 8 real bugs. Staged repair verified 5 of 16 runs for $0.32 in total, about $0.06 per verified repair. One TorchIO fix differs from the upstream patch and still passes every hidden check.
- Not every bug needs the model. For 8 of 31 seeded bugs, a deterministic correction fixes the code with no model call, which cuts model tokens by 18% at the same 98% repair rate.
- Honesty is built in. Failed runs are on the demo page on purpose. A script re-derives the numbers we quote from the raw results. The whole project cost about $5 in tokens.
## What we learned
Nemotron repairs well when it is given a check it can trust. Without a gate, the same model rewrote correct code and guessed conventions. A better prompt didn't fix that. A better check did.
We were careful not to oversell the oracle. A strong hand-written landmark test suite reached 97%, almost the same as the oracle's 98%, and that's an honest result: any check that can see physical positions helps. What the contract adds is that it's four fields written once per pipeline, not ninety lines of custom tests, and it's the only kind of check that can say "the convention is unknown". That's why, with it, the agent never guessed.
Inside real libraries, finding the bug turned out to be the easier half. The trace puts the right file in its top ten every time. Turning that location into a correct edit is still hard when the fix needs a real understanding of the algorithm, such as MONAI's offset computation or nibabel's axis assignment.
And the lesson we'll carry forward: verify the verifier. Every time we trusted a check without testing it, it found a way to be wrong.
## What's next for Spatial Contract
- Train on the oracle's signal. The millimetre error is a verifiable reward, which makes it a natural signal for reinforcement fine-tuning of a repair model.
- Close the known blind spot. Contract properties beyond world position, such as grid centring and flip-equivariance, would catch bugs like TorchIO #1310, which keeps data and affine consistent while choosing the wrong grid.
- Wider coverage: 4D series, DICOM input, and multi-input registration pipelines.
- More real bugs, from libraries such as dipy and ANTsPy, plus a unit-test baseline written by someone other than us.
- Lower cost per fix: Nemotron Nano for triage and Super for patching, so library runs get cheaper.
- Easier adoption: a pip release, and a pre-commit hook that checks every imaging pipeline in a repository.
Log in or sign up for Devpost to join the conversation.