Inspiration
Scientific software often sits in an uncomfortable gap between a paper and a product. A paper may clearly state its governing equations, assumptions, units, and boundary conditions, while the accompanying MATLAB or Python repository expresses the same ideas through hundreds of functions, parameters, and plotting scripts. Even when the code runs, it can be difficult to answer a more important question: does the implementation actually represent the model described in the research?
We encountered this problem while working with thermo-mechanical energy-pile simulations. Reproducing one result required tracing equations through numerical routines, checking experimental data formats, reviewing units and sign conventions, and manually comparing generated plots. Existing code assistants can explain individual files, but they rarely provide a testable chain of evidence connecting a scientific claim to the code and the final result.
That inspired ProofBench: a reproducibility agent that treats equations, source code, tests, and outputs as parts of the same system.
What it does
ProofBench accepts three primary inputs:
- A scientific document or model specification.
- A MATLAB or Python repository.
- Optional sample or experimental data.
It then builds an evidence graph connecting:
scientific claim → equation/assumption → implementation → executable test → result
GPT-5.6 extracts structured information from the research material, including variables, units, assumptions, boundary conditions, and expected qualitative behavior. Codex inspects the repository, finds the corresponding functions and parameters, and generates tests that can be executed against the real implementation.
ProofBench focuses on three kinds of checks:
- Consistency checks: variable names, units, dimensions, parameter ranges, and data schemas.
- Invariant and boundary checks: known limits, conservation or equilibrium conditions, zero-input behavior, and sign conventions.
- Behavioral checks: whether controlled changes in an input produce the scientifically expected trend in an output.
For a nonlinear model written abstractly as
[ mathbf{R}(mathbf{u}, oldsymbol{ heta}) = mathbf{0}, ]
ProofBench can trace the state vector (mathbf{u}), parameters (oldsymbol{ heta}), and residual calculation to their implementations, then verify convergence and expected limiting behavior using executable tests.
When a check fails, ProofBench does not simply label the repository as “incorrect.” It shows the source evidence, the relevant code, the executed test, and the observed result. It can then propose a patch and rerun the affected checks. The final output is an inspectable reproducibility report rather than an unexplained AI score.
Our initial benchmark uses a thermo-mechanical energy-pile model with MATLAB routines, experimental temperature data, and validation plots.
How we built it
The prototype is organized as a small end-to-end pipeline:
- Document ingestion: extracts text, equations, tables, variables, and nearby source context from PDF or Markdown material.
- Structured reasoning with GPT-5.6: converts the extracted material into typed claims such as equations, units, assumptions, boundary conditions, and expected trends.
- Repository analysis with Codex: maps those claims to files, functions, parameters, and output artifacts in a MATLAB or Python project.
- Test generation: produces human-readable, executable checks instead of relying only on natural-language review.
- Sandboxed execution: runs selected tests and captures logs, numerical results, and generated artifacts.
- Evidence reporting: presents each conclusion together with its document source, code location, test definition, and execution result.
We used Codex throughout the development workflow to explore the existing scientific repository, identify interfaces between legacy MATLAB components, scaffold the application, generate tests, review failure modes, and improve setup documentation. GPT-5.6 is used inside the product for the cross-document reasoning that connects scientific language and mathematical notation with implementation details.
A key design decision was to keep deterministic tools in the verification loop. GPT-5.6 proposes structured claims and tests, but parsers, test runners, and numerical comparisons produce the final evidence.
Challenges we ran into
The hardest challenge was translating scientific statements into tests without pretending that every sentence has one exact computational interpretation. Papers often omit implementation details, overload symbols, or describe expected behavior qualitatively. We addressed this by preserving the original source context and attaching a confidence level and rationale to every generated claim.
A second challenge was mapping mathematical notation to code. A symbol in a paper may appear under a different name, unit system, array layout, or abstraction in the repository. ProofBench therefore treats mappings as evidence-backed hypotheses that users can inspect and correct.
Execution safety and reproducibility were also important. Scientific repositories may depend on local paths, proprietary software, large datasets, or undocumented environment assumptions. We limited the first prototype to controlled MATLAB/Octave and Python workflows with explicit sample data and clear execution boundaries.
Finally, we had to resist building a general-purpose research platform. For Build Week, we narrowed the experience to one compelling loop: find a model claim, locate its implementation, generate a test, run it, and explain the evidence.
Accomplishments that we're proud of
We are proud that ProofBench turns a broad concern—“can this research code be trusted and reproduced?”—into a concrete workflow with inspectable artifacts.
In particular, we:
- Defined a traceable representation that connects scientific claims to code and runtime evidence.
- Assembled a realistic benchmark from an existing energy-pile research workflow rather than relying on a toy calculator.
- Designed generated tests to remain readable and editable by researchers.
- Kept deterministic execution at the center of verification instead of asking users to trust an AI-generated score.
- Created a focused demo path that shows both a failure and the effect of a proposed fix.
The result is not intended to replace peer review or scientific judgment. It gives reviewers and researchers a faster way to locate the parts of a computational result that deserve their attention.
What we learned
We learned that reproducibility is not a single pass/fail property. It is a chain of smaller claims, and each link needs different evidence.
We also learned that LLM reasoning becomes much more useful when its output is converted into executable, falsifiable artifacts. A generated explanation may sound convincing; a generated test that runs against the actual repository can be inspected, challenged, and improved.
Working with legacy scientific code reinforced the importance of explicit interfaces, sample data, deterministic commands, and clear README instructions. These are not merely packaging details—they are part of the scientific evidence.
Most importantly, we learned that the best role for GPT-5.6 in this workflow is not to declare whether a model is correct. Its role is to connect information across documents and code, surface hidden assumptions, and help formulate checks whose results can be independently verified.
What's next for ProofBench
Next, we plan to:
- Complete the energy-pile benchmark and publish a fully reproducible demo repository.
- Add an interactive view for accepting or correcting equation-to-code mappings.
- Support richer LaTeX equation parsing and table-aware parameter extraction.
- Expand sandboxed runners for additional scientific Python and MATLAB projects.
- Add repository-level regression certificates that can run in CI.
- Compare results across two independent implementations of the same model.
- Export machine-readable provenance so reports can accompany papers, datasets, and software releases.
Our longer-term goal is for every important computational figure in a paper to have a navigable path back to the equation, implementation, data, and test that produced it.
