Receipts

• 

  Receipts in one frame: 12 of 67 unguarded "done" were false; 0 of the 66 fixes Receipts shipped were. The receipt is from a real run.

• 

  How it works: Nemotron 3 Super fixes on its own branch; a witness forked from the same checkpoint re-runs your tests; plain code decides.

• 

  A real run: the agent said it fixed the failing tests. Re-run in a clean sandbox, the target test still fails.

• 

  100 real SWE-rebench bugs picked by a pre-registered rule: 12 of the unguarded agent's 67 "done" claims were false (17.9%).

• 

  The witness is a Token Factory Sandbox forked from the checkpoint; Sandboxes' checkpoint and branch make that fork exact.

• 

  Attempt 1 rejected: one target test still fails. The gate sends that evidence back to the agent for its one appeal.

• 

  Attempt 2 verified. The receipt lists every check, and SWE-rebench's official grader, run separately, agrees: resolved.

• 

  The same 100 tasks: 12 false "done" without Receipts, 0 false fixes shipped with it, at $0.41 of Nemotron tokens per fix.

• 

  Token Factory usage for all of development; 87,379,659 Nemotron tokens went to the 100 tasks.

• 

  Real Token Factory Sandboxes operations: 482 sandbox branches and 4,253 recorded steps over the 100 tasks.

• 

  Another repo, one command: receipts fix on PyFlakes issue 335, verified in 1m 7s for $0.03 of Nemotron tokens.

Receipts is a coding agent that does not get a vote on its own work. NVIDIA Nemotron 3 Super writes the fix on Nebius Token Factory; a fresh sandbox the agent never touched re-runs your tests, and plain code decides whether the fix ships.

On 100 real bugs picked by a rule I committed before the first run, the unguarded agent said "done" 67 times and was wrong 12 times, by SWE-rebench's own grader. With Receipts, the same agent shipped 66 fixes and none of them were false, at $0.41 of Nemotron tokens per fix.

Video (2:57) · Receipts from the real runs · The pre-registered report · Code. The 100 receipts from the experiment re-run in three commands, with Docker and no account or API key.

Inspiration

Here is how one of my runs ended, on a WALinuxAgent bug. Nemotron wrote: "I have successfully implemented the fix for the failing tests by adding the two required classmethods..." In a clean sandbox the target test still failed, and SWE-rebench's grader marked the task not resolved.

Coding agents say "done, all tests pass", and whoever merges the fix either takes that on faith or re-runs everything. I wanted to know how often the faith is misplaced, so I measured it on 100 SWE-rebench tasks held out from development: 12 of the agent's 67 "done" were false (17.9%, 95% interval 10.6% to 28.7%). In 6 of those 12, the agent's own last test run was green, so a reviewer who trusted that run would have merged the fix. I built a checker that runs where the agent never was, and a receipt that records every command it ran.

What it does

You give Receipts a Python repository, a commit and a failing test you wrote. It gives back a verified fix with a receipt, or a rejection with the evidence. Here is one command on PyFlakes issue 335, run on Token Factory:

1. Nemotron 3 Super on Nebius Token Factory plans and writes the fix on its own branch, in a Token Factory Sandbox.
2. Receipts forks the checkpoint taken before the agent started into a clean witness sandbox, applies the agent's patch without any change to tests or test configuration, and runs your tests there.
3. A gate written in plain code decides: your failing test now passes, nothing that passed before fails, no test was removed or skipped, the agent edited no existing test or test configuration, and every step traces back to the checkpoint of your code, with the tested patch bound to its SHA-256 digest. If it rejects, its evidence goes back to the agent verbatim, once, and a fresh witness checks the second attempt.
4. You get a receipt: every command on every branch, with exit codes, digests and the image it started from, plus a pull request description that carries the receipt.

One run from the video, pyfilesystem2's "FTPFS should use MDTM": Nemotron's first fix changed fs/ftpfs.py and reported success. In the witness, one of the three target tests, test_copy_preserve_time, still failed, so the gate rejected the fix and sent that line back. The second fix added getmodified to the base filesystem class. In a new witness all three target tests passed, all 333 tests that passed before still passed, and no test file was touched, so the gate passed it, and SWE-rebench's grader agreed. Both times the agent said its tests passed while its own last run showed errors; the verdict comes only from the witness's test run. The run used 904,972 tokens, $0.28.

Limits: it needs a failing test written by a person, and today it handles Python repositories whose tests run under pytest, on Token Factory Sandboxes or locally with Docker. A patch that detects it is under test could still fool the gate; the README lists what it cannot catch.

How I built it

• Nemotron 3 Super on Token Factory. The agent works through five tools (read, edit, write, run, finish), thinking off, temperature 1.0. In a scripted tool-loop test it completed 10 of 10 loops, against 6 for Nemotron 3.5 Lightning and 3 for Nemotron 3 Ultra. The client counts tokens and dollars per run at Token Factory's listed prices.
• Token Factory Sandboxes. Each branch is a chain of immutable images, so forking the checkpoint taken before the agent started is exact, and the witness provably never saw the agent's environment. The catalogue already holds the SWE-rebench-V2 images (598 of my 600 candidates), so each task starts from its real image. Trusted tools, a pinned static busybox and my test runner, are injected per run at a fresh path and never read from the image. Over the 100 tasks that came to 482 sandbox branches, 4,253 recorded steps and 87,379,659 Nemotron tokens, with up to four tasks running at once.
• The gate. Plain Python over JUnit reports and diffs; no model is asked whether a fix worked. 26 attack tests in the repository try to fool it with shadow packages, quoted patch paths, edited specifications and hidden skips.
• The experiment. I pre-registered two runs, committing the rules, the sample and the analysis before each one, and graded every patch again with SWE-rebench-V2's official grader in a separate sandbox. Every record is in the repository, including my own mistakes, and receipts reverify re-runs any of the 100 receipts.

Challenges I ran into

The hardest part was closing the ways to fool the gate. My pilot shipped two false fixes. One exposed a hole in my test runner: the agent put a copy of the package at the repository root, where my runner found it and the official grader did not. The pre-registration made me publish that, fix the runner, and start again. The other came from the environment: an official test that needs the internet, which fails in a sandbox without network access. A review before v2 found three more holes (a quoted patch path, an editable specification file, and shipped bytes that were never checked against what was tested). Each is closed and has an attack test.

The platform had surprises too. Downloading a directory from a sandbox returns 422 ("Path is not a regular file"), and listings name entries relative to the listed directory, so my directory reads failed live until the executor switched to listing and fetching file by file. contree-sdk 0.3.6 returned 403 on catalogue calls that succeed over REST, so the executor talks to the REST API directly. And a time field named elapsed_time turned out to hold CPU time, so my early receipts recorded CPU seconds as each command's duration until the executor switched to its own clock.

Accomplishments that I'm proud of

• 0 false fixes among the 66 that Receipts shipped, by SWE-rebench's official grader, on 100 pre-registered tasks. The gate and the grader agreed on 138 of 141 attempts; in the other 3 the gate was stricter.
• $0.41 of model tokens per verified fix.
• The 100 receipts can be checked without trusting me: on your own machine with Docker and no API key, or on Token Factory in about two minutes.

What I learned

Most of the engineering is in the checker: which files the agent may touch, which tests count, and proving that the bytes you ship are the bytes you tested. The agent loop was the easy part.

Pre-registering cost me a restart and was worth it. It forced me to publish the pilot's false fixes and close the runner hole before the second run, which shipped none.

Agent loops are almost all input tokens. Over the whole project Nemotron 3 Super read 146 million input tokens and wrote 1.65 million, because every turn resends the conversation, so the price of input decides what an agent costs.

What's next

A GitHub Action that runs the gate on a pull request and posts the receipt as a check. Then other languages. The gate reads pytest's JUnit XML report, and Java, JavaScript (jest) and Go runners can write the same format, so a new language needs a runner adapter plus its own rules for test ids and protected files.

How Receipts meets the judging criteria

• Technological Implementation: in the 100-task experiment, Nemotron 3 Super on Token Factory made every model call and Token Factory Sandboxes ran every agent, witness and grading branch; a model-free gate with 26 attack tests; a pre-registered evaluation graded by an independent official grader.
• Design: the product is the receipt. Each run page shows what the agent said beside what the record shows, the receipt page carries a verdict stamp, and re-verifying takes three commands. (receipts from real runs)
• Potential Impact: maintainers and teams that merge agent-written code can accept a fix on evidence. receipts fix runs on a Python repository with a failing pytest test, and each verified run comes with a pull request description that carries its receipt.
• Quality of the Idea: a referee that runs where the agent never was, and a public, pre-registered measurement of how often agents' "done" is false. The other claim checkers I reviewed read the diff, read CI evidence, or re-run commands in the agent's own working tree. Receipts re-runs the tests in a fresh fork of the checkpoint taken before the agent started, closes the loop with one appeal, publishes receipts anyone can re-execute, and shows on 100 pre-registered tasks, against the official grader, that it works.

Credits

Video music: "Wallpaper" by Kevin MacLeod (incompetech.com), licensed under Creative Commons: By Attribution 4.0 (https://creativecommons.org/licenses/by/4.0/).

Built With

• docker
• nebius-token-factory
• nemotron-3-super
• nvidia-nemotron
• pytest
• python
• swe-rebench
• token-factory-sandboxes