Receipts

The coding agent that isn't allowed to grade its own homework: Nemotron writes the fix, a fresh sandbox and plain code decide, and each fix comes with a receipt of every command.

Comment

Receipts is a coding agent that does not get a vote on its own work. NVIDIA Nemotron 3 Super writes the fix on Nebius Token Factory; a fresh sandbox the agent never touched re-runs your tests, and plain code decides whether the fix ships.

On 100 SWE-rebench bugs picked by a rule I committed before the first run, the unguarded agent said "done" 67 times and was wrong 12 times, by SWE-rebench's own grader. With Receipts, the same agent shipped 66 fixes and none of them were false, at $0.41 of Nemotron tokens per fix.

Receipts in one frame: 12 of 67 unguarded "done" claims were false and 0 of the 66 fixes Receipts shipped were, beside a receipt stamped VERIFIED

Video (2:57) · Receipt viewer · The pre-registered report · Code. Any of the 100 experiment receipts can be re-verified in three commands, with Docker and no account or API key, and a public GitHub Actions run re-verified all 100.

For judges

Try it live: the live page runs Receipts on one of five preset tasks from the experiment, with the judge code in the testing notes. A run takes one to five minutes and lists every step with the model or tool that did it.

  • Technological Implementation: one run, every command. Nemotron 3 Super on Token Factory writes each fix with five tools in a Token Factory Sandbox, and a second sandbox, forked from the checkpoint taken before the agent started, runs the tests again. Over the 100 tasks: 482 sandbox branches and 4,253 recorded steps.
  • Design: the receipt viewer. Its first screen gives the result and one run from the experiment as a log: claimed done, rejected, appealed, verified. Re-verifying any of the 100 receipts takes three commands, with Docker and no key.
  • Potential Impact: the False-Done Report. For anyone who merges fixes a coding agent wrote: 12 of the agent's 67 "done" were false, and 6 of those came with a green test run of its own. Receipts shipped 0 false fixes out of 66.
  • Quality of the Idea: the AERead trial. The agent never grades its own work and every rule was committed before the run it governs. On bug fixes from a project outside any benchmark, I measured where passing the tests stops being enough.

Limits

  • It needs a failing test written by a person, and a fix is only as good as that test. On AERead, 5 of the 13 patches Receipts verified were wrong and 3 were partial, although every one of them passed its target tests.
  • R8, the independent Nemotron review, only advises. A rule I fixed in advance allowed it to reject at most 3 of the 66 fixes the official grader accepts; it rejected 10.
  • Python repositories tested with pytest only. Tests that need the network cannot pass in a sandbox.
  • A patch that detects it is under test could fool any test-based gate, the official grader included.
  • The live page runs only five preset tasks, one at a time, with a daily limit and a spending cap. On your own repository, re-verifying needs Docker and no key, and fixing needs a Token Factory key.

Inspiration

Here is how one of my runs ended, on a WALinuxAgent bug. Nemotron wrote: "I have successfully implemented the fix for the failing tests by adding the two required classmethods..." In a clean sandbox the target test still failed, and SWE-rebench's grader marked the task not resolved.

Coding agents say "done, all tests pass", and whoever merges the fix either takes that on faith or re-runs everything. I wanted to know how often the faith is misplaced, so I measured it on 100 SWE-rebench tasks held out from development: 12 of the agent's 67 "done" were false (17.9%, 95% interval 10.6% to 28.7%). In 6 of those 12, the agent's own last test run was green, so a reviewer who trusted that run would have merged the fix. I built a checker that runs where the agent never was, and a receipt that records every command it ran.

What it does

You give Receipts a Python repository, a commit and a failing test you wrote. It gives back a verified fix with a receipt, or a rejection with the evidence. Here is one command on PyFlakes issue 335, run on Token Factory:

receipts fix on PyFlakes issue 335: the five checks pass and the run ends VERIFIED in 1m 7s for $0.03 of Nemotron tokens

  1. Nemotron 3 Super on Nebius Token Factory plans and writes the fix on its own branch, in a Token Factory Sandbox.
  2. Receipts forks the checkpoint taken before the agent started into a clean witness sandbox, applies the agent's patch without any change to tests or test configuration, and runs your tests there.
  3. A gate written in plain code decides: your failing test now passes, nothing that passed before fails, no test was removed or skipped, the agent edited no existing test or test configuration, and every step traces back to the checkpoint of your code, with the tested patch bound to its SHA-256 digest. If it rejects, its evidence goes back to the agent verbatim, once, and a fresh witness checks the second attempt.
  4. You get a receipt: every command on every branch, with exit codes, digests and the image it started from, plus a pull request description that carries the receipt.

Which model does what: Nemotron 3 Super writes the fix. A Token Factory Sandbox and plain Python decide, with no model. After a patch passes, two more Nemotron 3 Super calls review it, and their objections go on the receipt without changing the verdict.

Your failing test goes to Nemotron 3 Super on Nebius Token Factory, which works on an agent branch; a Token Factory Sandbox forked from the same checkpoint is the witness; plain-code gate; receipt. SWE-rebench's official grader runs separately.

One run from the video, pyfilesystem2's "FTPFS should use MDTM": Nemotron's first fix changed fs/ftpfs.py and reported success. In the witness, one of the three target tests, test_copy_preserve_time, still failed, so the gate rejected the fix and sent that line back. The second fix added getmodified to the base filesystem class. In a new witness all three target tests passed, all 333 tests that passed before still passed, and no test file was touched, so the gate passed it, and SWE-rebench's grader agreed. Both times the agent said its tests passed while its own last run showed errors; the verdict comes only from the witness's test run. The run used 904,972 tokens, $0.28.

Beyond the benchmark

The 100-task result is graded by SWE-rebench's official grader. I also wanted to see Receipts on a project outside any benchmark, so I ran it on bug fixes from AERead, an open-source project. I contribute to AERead; every fix in the pool was written by another contributor, and I wrote each issue text from the fix commit's message. As before, the pool, the order and every rule change were committed before the runs they govern (plan, results).

On two held-out sets of 8, Receipts verified 13. I read every verified patch against the upstream fix, and Codex read them separately; we agreed on all of them: 5 correct, 3 partial, 5 wrong. Every wrong or partial patch passes the target tests, which come from the upstream fix. One breaks a relative path that used to work; another refuses a provider's valid zero cost. Receipts checks that those tests pass in a clean sandbox, and in this trial passing them often left part of the issue unfixed. The regression check here ran only the fix's changed test files, because AERead's full suite does not finish in the 600-second limit. One false verification came from a bug in my screening script; the trial found two such bugs, and both are published with their effect on every task.

Changes I tested:

  • R7 checks for shortcuts, such as a helper nothing calls or a stub. In new runs on the original eight tasks, with R7 and a stricter target rule, 3 of the 4 tasks whose first patch was wrong were rejected instead; one task that had been fixed correctly got a wrong patch.
  • R8 is two separate Nemotron calls that can only object. I fixed a rule in advance: if it vetoed more than 3 of the 66 fixes from the 100-task experiment that the official grader accepts, it would ship advisory. It vetoed 10, mostly where a patch did not follow wording the issue suggested, so it ships advisory, and both reviews' objections are recorded on the receipt. On the second held-out set, reviewing after the runs and changing no verdict, it flagged 2 of the 4 patches that are not correct and neither correct one.

How it differs from similar entries

Several entries check or pick a coding agent's fix. From their public READMEs, read on October 26:

  • FixedYet reproduces a GitHub issue with a test Nemotron writes and decides by rules over exit codes. Its README reports 36 of 45 reproductions on 15 SWE-bench Verified issues and says they were not held out.
  • forkfix branches the agent over Sandbox snapshots, and a Nemotron 3 Super judge picks the patch: 15 of 79 held-out SWE-bench Lite tasks solved, against 4 for a single attempt.
  • FixFork races three fix hypotheses for a failing test and keeps the branch whose test command goes green with the fewest changed lines. It reports case studies, including a fix merged in tomlkit after it was finished by hand.
  • Polygraph checks a pull request's claims by running the base and head code side by side, with Nemotron 3 Ultra ruling on each claim.
  • slash-agent plans with Nemotron 3 Ultra, sends the writing to the cheapest Nemotron its quality floor allows, and returns a receipt of what it spent. That receipt is a bill; mine records the commands that tie a patch to the test run that passed it.

What Receipts adds: the test that decides is yours and runs in a sandbox forked from before the agent started; by default no model takes part in the verdict; a rejected fix gets one appeal with the gate's evidence; and the result is measured as a false-done rate on a sample fixed in advance.

How I built it

  • The agent is Nemotron 3 Super on Token Factory, working through five tools (read, edit, write, run, finish), thinking off, temperature 1.0. In a scripted tool-loop test it completed 10 of 10 loops, against 6 for Nemotron 3.5 Lightning and 3 for Nemotron 3 Ultra. The client counts tokens and dollars per run at Token Factory's listed prices.
  • In Token Factory Sandboxes each branch is a chain of immutable images, so forking the checkpoint taken before the agent started is exact, and the witness provably never saw the agent's environment. The catalogue already holds the SWE-rebench-V2 images (598 of my 600 candidates), so each task starts from its own SWE-rebench image. Trusted tools, a pinned static busybox and my test runner, are injected per run at a fresh path and never read from the image. Over the 100 tasks that came to 482 sandbox branches, 4,253 recorded steps and 87,379,659 Nemotron tokens, with up to four tasks running at once.
  • The gate is plain Python over JUnit reports and diffs; no model is asked whether a fix worked. 26 attack tests in the repository try to fool it with shadow packages, quoted patch paths, edited specifications and hidden skips.
  • For the experiment I pre-registered two runs, committing the rules, the sample and the analysis before each one, and graded every patch again with SWE-rebench-V2's official grader in a separate sandbox. Every record is in the repository, including my own mistakes, and receipts reverify re-runs any of the 100 receipts.

The Token Factory console's usage page for all of development, with Nemotron 3 Super's 146.41M input and 1.65M output tokens highlighted; 87,379,659 Nemotron tokens went to the 100 tasks

Challenges I ran into

The hardest part was closing the ways to fool the gate. My pilot shipped two false fixes. One exposed a hole in my test runner: the agent put a copy of the package at the repository root, where my runner found it and the official grader did not. The pre-registration made me publish that, fix the runner, and start again. The other came from the environment: an official test that needs the internet, which fails in a sandbox without network access. A review before v2 found three more holes (a quoted patch path, an editable specification file, and shipped bytes that were never checked against what was tested). Each is closed and has an attack test.

The platform had surprises too, all written up in my feedback: downloading a directory from a sandbox returns 422, contree-sdk 0.3.6 returned 403 on catalogue calls that succeed over REST, and a field named elapsed_time holds CPU time, so my early receipts recorded CPU seconds as durations until the executor switched to its own clock.

Accomplishments that I'm proud of

  • 0 false fixes among the 66 that Receipts shipped, by SWE-rebench's official grader, on 100 pre-registered tasks. The gate and the grader agreed on 138 of 141 attempts; in the other 3 the gate was stricter.
  • The 100 receipts can be checked without trusting me, with Docker and no API key, and a public GitHub Actions run re-verified all of them.

What I learned

Most of the engineering is in the checker: which files the agent may touch, which tests count, and proving that the bytes you ship are the bytes you tested. The agent loop was the easy part.

Pre-registering cost me a restart and was worth it. It forced me to publish the pilot's false fixes and close the runner hole before the second run, which shipped none.

Agent loops are almost all input tokens. Over the whole project Nemotron 3 Super read 146 million input tokens and wrote 1.65 million, because every turn resends the conversation, so the price of input decides what an agent costs.

What's next

A GitHub Action that runs the gate on a pull request and posts the receipt as a check. Then other languages. The gate reads pytest's JUnit XML report, and Java, JavaScript (jest) and Go runners can write the same format, so a new language needs a runner adapter plus its own rules for test ids and protected files.

Credits

Video music: "Wallpaper" by Kevin MacLeod (incompetech.com), licensed under Creative Commons: By Attribution 4.0 (https://creativecommons.org/licenses/by/4.0/).

Built With

  • docker
  • nebius-token-factory
  • nemotron-3-super
  • nvidia-nemotron
  • pytest
  • python
  • swe-rebench
  • token-factory-sandboxes
Share this project:

Updates

Submission history