Paper Reproducibility Auditor

An AI referee that checks every number in a paper and re-runs its results. It found a mistake in "Attention Is All You Need".

Comment 1

Inspiration

Peer review is the main filter science has, and it is overloaded. A referee has a few hours for a paper: enough to judge the idea, rarely enough to check every number against every other number, and almost never enough to re-run the results. So mistakes get through, and "can this be reproduced?" is often answered by nobody.

Language models can read a paper in seconds, but a model's opinion is not evidence. I wanted to build a referee whose every finding comes with its proof: the passage it quotes, the arithmetic it did, or the code it ran.

What it does

Give it an arXiv id or a PDF. In about a minute it writes a referee report:

  • A review: summary, strengths, major and minor concerns, a reproducibility checklist, questions for the authors. Every concern must quote the paper, and every concern is argued against before it is kept.
  • Number checks done by code: table averages and totals, percentages against their counts, every number of the abstract against the body, and results printed twice with two values.
  • Related work: the closest papers found on the web, and whether the paper cites them.

For computational papers it goes further: it writes code from the paper's equations, runs it, and compares the output with the paper's tables digit by digit.

On well-known papers it found:

  • Attention Is All You Need: the English-French BLEU of the big model is 41.8 in the abstract and 41.0 in the results section.
  • Adam: the convergence theorem assumes a decaying step size and momentum, while the algorithm as written keeps them constant:

$$\alpha_t = \frac{\alpha}{\sqrt{t}}, \qquad \beta_{1,t} = \beta_1 \lambda^{t-1}$$

  • A 2025 numerical-methods paper: all eight results reproduced from its equations, two of them only with a mass m = 1 that the paper never states. Its scheme depends on the mass, for instance through

$$c = \frac{h}{m}\left(1 - \frac{2}{15}h^2\omega^2 + \frac{1}{300}h^4\omega^4\right)$$

and with m = 0.5 or m = 2 the paper's table does not come out.

How I built it

The rule that runs through the whole system: models propose, code checks.

Every model call goes in real time to Nebius Token Factory through its OpenAI-compatible API:

Model Role
NVIDIA Nemotron 3 Ultra writes the review, challenges each concern, plans which results to re-compute
NVIDIA Nemotron 3 Super writes the reproduction code, and acts as the independent second reader
NVIDIA Nemotron 3 Nano quick tasks: search queries, classifying why a script failed
MiniCPM-V reads figures cut out of the PDF, transcribes PDFs without a LaTeX source

Tavily searches for related work and for details a paper defers to another one.

Everything a model says is checked by code before it reaches the report:

  • Quotes are looked up in the paper's text. A concern whose quote is not there is dropped.
  • Arithmetic is never done by a model. A table average is recomputed from the cells, and a printed input such as 41.8 is taken to stand for anything in [41.75, 41.85), so rounding is never reported as an error. A mismatch found by matching words is reported only if a second model, which did not set it up, confirms both numbers are the same quantity.
  • Verdicts follow fixed rules. A re-computed value matches the paper's value when

$$|\hat{x} - x| \le \max\left(10^{-5}\,|x|,\ u(x)\right)$$

where u(x) is the last digit the paper prints. Confidence is high only when the agreeing significant digits add up to at least six, under no assumption.

  • Sensitivity: each detail the paper omits becomes an assumption with alternatives, and every result is re-run with each one changed. That is how the missing mass was found.

The backend is Python with a FastAPI server. PDFs are handled with PyMuPDF and arXiv LaTeX sources are parsed directly. The web page is hand-made HTML and CSS. The public demo runs on Render with a spending cap and per-visitor limits.

Challenges I ran into

  • Models invent problems. Early reviews complained about a "missing Table 1" that was there, or "duplicated sections" that appeared once. Requiring a verifiable quote, a challenge round, and code checks for facts code can decide (undefined references, duplicates) removed these.
  • False mismatches. A model compared a dev-set score with a test-set score, or dropped a factor of 2 in a parameter count. The two-question confirmation ("same quantity?", "computed as the paper defines it?") and a rule for round-factor discrepancies stopped them without losing real errors.
  • Non-determinism. The same request at temperature 0 with a fixed seed returned different text almost every time. I record every model call, search and script run under a SHA-256 hash of its exact input (model, messages and parameters), so the same paper always gives the same report, byte for byte.
  • Errors disguised as answers. On long inputs the endpoint sometimes returned a timeout message as if it were the model's reply. These are now detected, retried and never recorded.
  • Quotes that don't match. LaTeX accents, ligatures, ellipses and PDF line breaks meant a correct quote often failed an exact search. Quote matching now normalises text and accepts quotes cut with "…".
  • Budget. Token Factory has no server-side spending limit, so the app keeps its own ledger, stops at a cap, and the public demo serves stored reports for free while limiting new audits.

Accomplishments that I'm proud of

  • To measure the checks, I planted 133 errors in copies of 11 real papers: wrong averages and totals, abstract figures the body never prints, percentages that don't match their counts, broken references. The checks caught 132. On the unmodified papers they raised only two flags, and both are real problems of those papers.
  • A full re-computation of a numerical-methods paper from its equations, matching its table to 17 significant digits.
  • Every finding in a report links to its evidence: the quote, the arithmetic, or the code, which anyone can download and run.

What I learned

  • A model is far more useful as a proposer than as a judge. The quality jumped when every claim a model made had to pass a check written in code.
  • A second, independent model is a cheap and effective filter, as long as it does not see the first model's reasoning.
  • Planting errors is the honest way to measure a checker: it gives a number for what is caught, and running on clean papers gives a number for false alarms.
  • Reproducibility is mostly about missing details. The most common finding is not "the paper is wrong" but "the paper does not say".

What's next

  • Running generated code in managed sandboxes instead of a local process.
  • Reading result values off figures, not only describing them.
  • A mode for authors: run the auditor before submitting and fix what it finds.

Built With

  • arxiv
  • css
  • fastapi
  • html
  • javascript
  • latex
  • minicpm-v
  • nebius-token-factory
  • nemotron-3-nano
  • nemotron-3-super
  • nemotron-3-ultra
  • nvidia-nemotron
  • playwright
  • pymupdf
  • python
  • render
  • tavily
  • uvicorn
Share this project:

Updates

Submission history