Inspiration

A paper's headline number is rarely re-run by anyone. Reproducing it means finding the code, guessing the environment, running it several times and working out why the result is different, which takes hours or days. So most claims are simply trusted. We wanted an agent that does that work and, just as importantly, tells you honestly when it could not check something.

What it does

You paste an arXiv link. Receipts:

extracts the paper's headline claim,
searches for the code repository, datasets, errata and prior replication attempts,
writes a reproduction script and runs it over many seeds in a sandbox,
compares the claim with the distribution of results and returns a verdict: reproduced, partially reproduced, discrepancy found, or inconclusive,
explains why a number differs, and
saves a receipt: the executed code, per-seed numbers, logs, a tamper-evident hash and a one-command re-run.

Honesty is the feature. If the runs break, the answer is "inconclusive", never an accusation. A run that was shrunk to fit the budget can never count against a paper. Scripts that hard-code the claimed number are rejected.

How we built it

Every model call goes through Nebius Token Factory to an NVIDIA Nemotron 3 model:

Nemotron 3 Super extracts claims, writes the reproduction code, and repairs it from the error log.
Nemotron 3 Nano summarises job logs and filters search results so that only pages that really report an erratum or replication of the paper are shown.
Nemotron 3 Ultra reasons about why a number differs and names one cause, grounded in the computed numbers and in any variant of the protocol that matches the claim.
Tavily finds the repo, data, errata and replication attempts at runtime (details below).

How we use Tavily

Search is load-bearing in Receipts, not decoration: without it the agent often has no code to run, no data to run it on, and no context for the verdict. For every paper Tavily runs four searches:

official code, for papers that do not link their repository,
errata and corrections, so a known problem is shown next to the verdict,
replication attempts by other people,
the claim's dataset by name, so the reproduction script gets real download links instead of guessing URLs.

Raw hits are not trusted. Nemotron 3 Nano keeps only pages that genuinely report an erratum or replication of this paper, and the paper's own pages are never counted as evidence about itself. Receipts also reads the repository README for the authors' own notices.

Example: on FITS (ICLR 2024), the receipt shows the authors' notice about a long-standing test-loader bug and the ETTh1 dataset repository that Tavily found, right next to the verdict that the corrected code still reproduces the headline. That is the context a reader needs to trust the number, gathered automatically.

Reproductions run in locked-down containers (all capabilities dropped, resource limits). The statistical verdict compares the claim with the distribution over seeds. The same sandbox image can run as a Nebius Serverless Job.

Challenges

Nemotron 3 models reason before answering; with small token budgets the answer came back empty. We budget for reasoning tokens and retry with a bigger budget.
Preventing false accusations took more work than detecting errors: broken environments, shrunken runs and generic search hits all had to be handled so the agent never blames a paper for our own limitations.
Real repositories break in ordinary ways. Getting our first real paper through end to end exposed a dozen general problems: version pins that no longer install, Docker's 64 MB shared memory crashing PyTorch, launch scripts lost in repositories full of log files, and Windows encoding errors. The worst was a script that failed, printed -1 as a placeholder, and was counted as "reproduced" because lower error is better. Receipts now rejects impossible values and treats "far better than the paper" as suspicious. Every fix applies to every paper, not just that one.
Our hackathon account could not create Nebius Serverless Jobs: every job create, even a dry run with a public CPU image, returned PermissionDenied. Nebius support traced it to AI Cloud billing not being activated on our tenant, which the Token Factory credits do not cover. The Nebius runner is implemented but not exercised end to end; the Docker runner uses the identical image.

Accomplishments

A real ICLR paper reproduced end to end. For FITS (ICLR 2024 Spotlight), whose authors had disclosed a test-loader bug, Receipts found the bug notice, ran the corrected official code with the authors' own settings over 5 seeds, and confirmed the ETTh1 headline: 0.3721 ± 0.0001 against the claimed 0.372. Receipt
A known-answer benchmark of 8 cases: 4 planted bugs (feature-selection leakage, hyperparameters tuned on the test set, training accuracy reported as test accuracy, duplicate rows) were all detected, 0 of 2 clean controls were wrongly flagged, and a proprietary-data case correctly came out inconclusive.
A real paper ("Attention Is All You Need") went through the whole pipeline and was correctly reported as inconclusive, because its 2017 TensorFlow 1.x code cannot be installed today, instead of being flagged.
Tamper-evident, re-runnable receipts.

What we learned

Deciding not to give a verdict is as important as giving one. Planted-error tests with known answers are the honest way to measure an agent like this. And an agent that grades other people's results has to distrust its own: most of our real-paper work went into making sure a broken run could never look like a success.

What's next

Run it on 100+ real papers with public code and publish the success rate: how often it reaches a verdict, and how often that verdict is right.
A reproducibility pre-screen before peer review: authors self-check before submitting, and editors get a receipt instead of a promise that the code works.
Parallel jobs on Nebius GPUs once AI Cloud billing is active, and a public leaderboard of receipts.