Project Activity
Manage Connected Sources
Result
Recent History
Start Investigation
Review
Review result
Dashboard
Review the Change
Welcome Page
Sign In

Inspiration

A dependency upgrade can leave a scientific program running successfully while changing the result that matters.

Most dependency tools help identify outdated packages or broken APIs. We wanted to investigate a quieter problem: what happens when the code still runs, but its numerical conclusions change?

NumPy’s changes to scalar promotion provided a concrete starting point. A small difference in intermediate precision can move a result across a decision threshold. ReproDelta grew from the idea that maintainers should be able to inspect an executable counterexample, understand the tested scope, and reproduce the observation before accepting an upgrade.

What it does

ReproDelta investigates numerical behavior across two Python environments.

The workflow starts with a contract: valid inputs, the measurement of interest, a decision rule, numerical tolerances, and an execution budget. It then runs the same inputs in both environments, checks repeatability, and searches for a valid input whose declared outcome changes.

For example, a decision might depend on:

$$ r > 0.5 $$

An old environment returning approximately (0.500000015) and a new environment returning exactly (0.5) can produce different outcomes under an explicitly defined tie policy.

ReproDelta records the evidence behind that difference. Its curated investigation can simplify a witness, compare numerical checkpoints, test a targeted intervention, and verify a constrained repair against frozen holdouts.

Connected GitHub and GitLab changes receive dependency reviews and eligibility checks. A gated approved-project pilot adds reviewed project profiles, isolated execution, bounded search, private replay exports, and redacted sharing.

How we built it

We built the application with Python, FastAPI, PostgreSQL, server-rendered templates, and JavaScript.

The investigation engine separates input validation, paired execution, search, reduction, verification, and reporting. Source snapshots, contracts, inputs, and environments are bound to cryptographic digests so findings remain tied to the exact experiment.

Real project execution uses separate workers with E2B and Oracle Docker adapters. A durable PostgreSQL queue manages leases, worker heartbeats, cancellation, and execution-cost reservations.

We integrated NVIDIA Nemotron through Nebius Token Factory for explanation and repair proposals in the curated workflow, alongside an optional numeric-summary hypothesis feature for approved-project findings. Model suggestions remain separate from measured results, and mock responses are explicitly labelled.

Challenges we ran into

The hardest challenge was deciding what a numerical difference actually establishes. Harmless rounding, a changed decision, an unstable execution, and an incorrect repair require different conclusions.

Repair verification was particularly demanding. Making two environments agree can still damage previously consistent results. We added measurement-preservation checks and kept verification inputs frozen before patch generation.

We also had to handle immutable source revisions, environment provenance, private-source transfer, interrupted workers, and meaningful progress reporting. These requirements shaped both the execution architecture and the interface.

Accomplishments that we're proud of

Building a complete curated investigation with real pinned NumPy environments.
Executing constrained source patches and rejecting an intentionally overfitted repair.
Producing downloadable, integrity-checked regression replays.
Reproducing an independently reported NumPy mask-shift issue at the expression level.
Adding repeated measurements, unchanged controls, and equal-budget search comparisons.
Building an approval-gated project execution path with durable jobs and explicit evidence boundaries.

What we learned

We learned that reproducibility requires more than recording a package version. Source identity, installed environments, input validity, repeatability, and decision semantics all matter.

We also learned to treat AI-generated explanations as testable hypotheses. Execution supplies observations; approved contracts and deterministic checks determine what those observations support.

A useful investigation must make uncertainty understandable, including when no witness is found or a stage cannot complete.

What's next for ReproDelta

We plan to validate the approved-project workflow across more independently sourced public and private projects, complete deployment and isolation checks, and expand beyond the current NumPy-focused boundary.

Further work includes stronger scientific reference checks, broader experiment strategies, and mechanism localization and repair verification for approved projects.

Our goal is to make dependency reviews more evidence-driven: a reproducible finding, a clearly stated scope, and an experiment another maintainer can run.