PeerProof

An executable peer reviewer that reproduces research workflows, independently stress-tests claims, and returns an auditable Evidence Ledger.

Comment

Inspiration

Computational research is often evaluated by reading papers and checking whether the reported conclusions sound plausible. But many important failures only become visible when the underlying code and data are actually executed.

PeerProof was inspired by a simple question: what would peer review look like if an AI reviewer could inspect the repository, run the submitted workflow, independently recompute the evidence, and test whether the conclusion survives small changes?

What it does

PeerProof is an executable AI peer reviewer for computational research.

It:

  • extracts structured scientific claims from a paper
  • investigates the accompanying code repository
  • identifies the intended executable workflow
  • applies deterministic policies before allowing repairs or execution
  • runs the submitted analysis locally
  • independently recomputes the reported statistics
  • performs robustness checks such as leave-one-out analysis
  • assigns a verdict such as Reproduced, Fragile, Failed, or Unverifiable
  • produces a downloadable Evidence Ledger containing the full verification trail

In the Lighthouse benchmark, PeerProof reproduces the reported result but finds that removing one observation changes the p-value from below .001 to .849, producing a Fragile verdict.

PeerProof also includes a public-evidence audit using the Datasaurus Dozen case. It independently matches 20 of 65 printed values from a pinned package and mirror snapshot, reports the mismatch transparently, and does not claim to have run the original publication pipeline or proven the paper false.

How we built it

PeerProof is built as a local Node.js application with a browser interface, command-line interface, deterministic policy engine, policy-governed execution of reviewed fixtures, independent statistical verification, and a machine-readable Evidence Ledger.

This Build Week submission executes two reviewed evidence packages; arbitrary uploaded-repository execution is intentionally disabled.

GPT-5.6 is used for structured scientific claim extraction in credentialed mode.

Codex is used at runtime to investigate repository structure, locate executable workflows, and produce structured repair proposals and RunPlans. These AI recommendations do not directly authorize code changes or execution. PeerProof applies separate deterministic policies before anything is modified or run.

Codex also accelerated development: it helped scope the MVP, trace provenance and policy gaps, implement verifier hardening, generate adversarial tests, and conduct clean-room release reviews. GPT-5.6 handles structured claim extraction; deterministic verifier code—not model prose—assigns verdicts.

The default judge workflow requires no API key. It uses clearly labeled reviewed offline AI fixtures so judges can evaluate the complete workflow without API credits. Policy approval, code execution, statistical recomputation, robustness testing, verdict generation, and Evidence Ledger creation still run locally.

The repository also includes Docker support, GitHub Actions CI, build-integrity manifests, runtime-closure validation, policy-bundle checks, and adversarial evaluation fixtures.

Challenges we ran into

One major challenge was separating AI recommendations from trusted execution. An AI model may suggest a plausible repair or command, but that suggestion should not automatically receive permission to modify or execute research code. We solved this by placing deterministic policy checks between investigation, repair, execution, and verification.

Another challenge was preserving evidence provenance. PeerProof records the application commit, governed runtime files, policy identities, evidence hashes, execution outputs, and independent recomputation results. During final release validation, we also corrected Git commit provenance so a fresh public clone reports its actual repository HEAD.

We also had to make the no-key judging experience honest. Rather than silently pretending fixture output was live model output, the interface and ledger clearly identify reviewed offline AI fixtures.

Accomplishments that we're proud of

  • A complete executable review pipeline rather than a text-only critique
  • Independent statistical recomputation instead of trusting author-generated output
  • Deterministic approval boundaries around AI-proposed repairs and execution plans
  • Robustness testing that can distinguish a reproduced result from a fragile one
  • A downloadable Evidence Ledger with traceable commit and evidence provenance
  • A no-API-key judge workflow that remains transparent about offline AI fixtures
  • 155 automated tests passing
  • More than 90% line coverage
  • Clean installation from the public GitHub repository with zero reported npm vulnerabilities
  • Successful GitHub Actions validation on the final public submission commit

What we learned

We learned that executable peer review requires more than connecting an AI model to a repository.

The difficult part is establishing trust boundaries: which files are approved, which data source is authoritative, which repairs are permitted, which commands can run, and which results must be independently recomputed.

We also learned that reproducibility and robustness are different. A result can be reproduced exactly and still be scientifically fragile. PeerProof therefore reports both whether the submitted result can be recreated and whether it survives predefined perturbations.

What's next for PeerProof

Next, we plan to expand PeerProof beyond the current reviewed statistical contracts by adding more verifier types, additional programming-language runtimes, richer robustness suites, and support for more research repository layouts.

We also plan to improve credentialed live-model workflows, signed release attestations, container-isolated execution, collaborative review reports, and integrations with research publishing and continuous-integration systems.

Built With

Share this project:

Updates