RERUN

RERUN is a reproducibility auditor that runs a paper's code in a Nebius Token Factory sandbox and signs what actually happened, with every number traced to a record.

Comment

Does this paper's code still run? RERUN finds out by running it.

Inspiration

Research code rots. The tempting fix is to let an LLM patch it until it runs. A patched run that "works" may have swallowed the error, faked the data or changed the computation, and nobody measures how often the repair is real. I wanted a tool that runs the code for real, says what stops it, and never claims a success it cannot prove.

What it does

You paste a paper's repository. RERUN pins it to an exact commit, rebuilds the environment of the paper's era deterministically, and runs the documented command in a Nebius Token Factory sandbox. When the run fails, fixed rules classify what stopped it and say who can remove it. NVIDIA Nemotron models work in narrow roles: Nano finds the entrypoint or abstains, Super proposes repairs, Ultra chooses between candidates and may only downgrade a verdict. A deterministic tamper gate vets every proposal, a cost guard stops what it cannot fund, and each run becomes a hashed Reproduction Passport anyone can verify offline. Tavily is called at runtime before each repair attempt; the references offered and the sources cited are stored on the attempt.

How I built it

  • Backend in Python (FastAPI), frontend in React and TypeScript.
  • Execution: every baseline run and re-execution happens in a Token Factory sandbox; the three Nemotron models are called through Token Factory's OpenAI-compatible endpoint, with per-call token usage stored in each record.
  • Method: every held-out set was registered before its draw and run once, never used to tune the system. A harness version is sealed only after a live verification of every sandbox-touching code path.
  • Every figure in the README and on this page is generated from committed records (reports/v1.9/figures.json); no number is typed by hand.

What I measured

Three held-out sets: 3 of 26 repositories ran their documented command (TEST 1 of 8, TEST-B 1 of 8, TEST-C 1 of 10); "ran", never "reproduced". On TEST-C, 7 of 9 non-running entries got an actionable diagnosis (6 judged strictly).

The benchmark: two cheat sets ship in benchmark/ with hashes and a scorer that reproduces every published table: 337 planted patches, and 166 cheats with 42 honest controls written by a separate agent that had not seen the checks (144 cheats measured). On the planted held-out half the full pipeline adopted 0 cheats but refused all 31 honest controls that passed the run, so that table has no summary sentence.

Challenges and limits, said plainly

  • Repair's yield is small: 2 of 24 exploratory gate entry-runs reached a RUNS_* verdict.
  • Anti-cheat is not a headline claim: the shipped harness adopted 14 of the 50 independent cheats aimed at repositories whose run really fails (28%).
  • Behavioural checks that refuse such patches also refused 16 of 27 honest controls, so they ship as REVIEW_REQUIRED: a signal that a human should look, not cheat detection. It also marks 5 of the 9 honest repairs harness-v1.9.0 adopted.
  • Tavily is offered and sometimes cited (24 of the 88 live runs that were offered references cite one); whether a citation shapes a decision is not measured.
  • The sandbox is CPU-only with no documented memory figure; GPU-only papers end as GPU_REQUIRED.

Accomplishments

203 live sandbox runs, 326 commits, a sealed harness, a published benchmark with the project's own losses in it, and a certificate format that verifies without trusting RERUN's own UI.

What's next

GPU sandboxes (a Nebius AI Cloud compute backend is already scaffolded), more failure classes, a new independent cheat set, and a mode that journals and artifact reviewers can plug into their own process.

Try it

Built With

Share this project:

Updates

Submission history