Repro

Inspiration

I was inspired by how many GitHub projects publish impressive performance claims that are difficult to verify independently. A README may report faster speed or higher accuracy, but the required data, dependencies, hardware, and evaluation method are often unclear. ReproCheck was created to turn those claims into a traceable investigation rather than accepting them at face value.

What I Learned

I learned that reproducibility is more than rerunning a command. It requires:

• Pinning the exact repository revision.
• Preserving the original wording and evidence.
• Separating documented facts from measured results.
• Recording missing dependencies and environmental limitations.
• Treating model-generated plans as untrusted until validated.

A useful way to think about measurement differences is:

Δ = m_measured - m_reported

However, the difference alone is not enough; the experiment’s method, dataset, baseline, and environment must also be understood.

How I Built It

ReproCheck uses a TypeScript/Fastify server and a lightweight JavaScript web interface.

1. The server accepts a public GitHub repository URL and fixes the investigation to a specific commit.
2. An AI-assisted pipeline extracts quantitative claims, checks their evidence, and designs experiments.
3. Optional online searches provide missing public sources or datasets.
4. Evidence is read through a local DSH sandbox, while feasible benchmarks run in isolated workspaces.
5. Generated plans, commands, outputs, observations, and reports are saved as structured JSON artifacts.
6. The frontend receives live progress through Server-Sent Events and displays claims, evidence, experiment results, and final assessments.

The system deliberately reports outcomes such as evidence_only, unsupported, and inconclusive instead of claiming success too easily.

Challenges

The main challenge was handling uncertainty. Some projects require unavailable GPUs, private datasets, external APIs, or credentials, making exact reproduction impossible. AI-generated plans also sometimes contain invalid claim references, unverifiable citations, or unsafe assumptions, so I added schema validation, quote matching, command checks, and limited retries.

Long-running experiments introduced another challenge: users need progress updates, cancellation, and partial reports even when execution fails. Finally, the local sandbox improves control but is not a perfect security boundary, so ReproCheck records its environmental limitations rather than pretending to provide guaranteed isolation.

Outcome

ReproCheck is not designed to produce a simplistic “verified” label. It provides a transparent trail from a project’s original claim to the available evidence, the attempted experiment, the measured result, and the remaining uncertainty.

Built With

• deepseek