LabRepair

Two patches. Same total. One is wrong. Check the evidence behind AI-repaired research code.

Comment

Inspiration

AI can patch a research program until its headline number looks right. But a matching score can conceal the wrong correction. For a research software engineer, that is a dangerous handover: the code appears repaired, while the evidence behind the result has changed.

LabRepair starts with a deliberately small quantum-analysis example. Two patches both produce 2.60. Only one agrees with all four registered correlations. A total-only check accepts the wrong patch; an evidence-level check rejects it.

What it does

LabRepair makes a proposed repair inspectable before it becomes a research decision. It binds the proposal to frozen source and policy, preserves protected counts and tests, recomputes the registered values, and exports a source-bound review record.

The public browser demo runs three fixed synthetic cases without an account or API key: the declared repair, a misleading same-total repair, and incomplete evidence. Reviewers can compare the pair values, inspect the candidate change, record a hold or review intent, and download the evidence. No browser action sends a model request, executes candidate Python, authorizes payment, or promotes a scientific claim.

The most important distinction is visible: the repair passed; the original 3.40 claim did not. Repair acceptance and scientific-claim acceptance are separate decisions.

How we built it

NVIDIA Nemotron 3 Nano, served through the Nebius Token Factory inference API, proposed two bounded parameter arrays. An operator-owned adapter validates those arrays and constructs an allowlisted source change. It does not accept arbitrary generated Python. The declared channel convention was supplied in the prompt; the model translated that instruction into parameters rather than discovering an unknown scientific convention.

A recorded disposable Nebius Sandbox operation then ran the baseline, the model-proposed repair, and an operator-authored negative control against unchanged protected tests. The repair passed 4/4 checks. The baseline and deceptive control failed as expected. The recorded operation completed in 1.273 seconds with guest networking disabled.

On 30 October, the repository's own live CLI made one additional Token Factory call: HTTP 200, 121 input tokens and 38 completion tokens, followed by deterministic review. That new call did not launch a new Sandbox. Exact request, response, transport and review records are included separately from the earlier cloud run.

Python 3.12 provides the local review and direct API client. The portable judge interface uses browser JavaScript with exact rational arithmetic and source bindings. A shared SQLite ledger limits our live CLI to five attempts and US$0.05 reserved per Taipei day, retaining US$0.01 for failed or uncertain attempts. This is an application control, not a Nebius account-wide billing cap.

Challenges

Matching a final total was easy; preserving the meaning of every underlying value was the real challenge. We built an explicit counterexample so a reviewer can see why the extra checks matter.

We also had to separate four kinds of evidence: a model response, an isolated code execution, an interactive local recomputation, and a human review intent. They have different authority. A saved cloud result is not a new execution, and a hash proves content consistency rather than a provider signature.

Accomplishments

  • A concrete counterexample that makes a misleading repair visible in seconds.
  • Real NVIDIA inference on Nebius and a recorded before/after/negative-control Sandbox run.
  • A direct runtime API client in the repository, with bounded inputs, no automatic retries, and persistent attempt reservations.
  • A key-free interactive review and downloadable evidence.
  • 61 product, mocked-transport and budget checks, plus 18 portable engine/DOM checks, passing in the recorded release increment. These are fixed-case engineering checks, not a general accuracy benchmark.

For Windows with Python 3.12, the product verification entry point is python scripts/verify_poc.py; it makes no provider call. Optional Node.js checks are separate. The full legacy security suite requires the reviewed Linux environment; no macOS verification is claimed. The local verifier's six curated offline fixtures differ from the three public browser cases and the three historically executed Sandbox variants. Their counts are not combined into an accuracy score.

What we learned

Giving a model a small, well-specified job does not remove the need for independent verification. Nemotron can propose the correct parameters while the original research claim still remains unsupported. A useful developer tool must preserve that distinction through execution, review, and handover.

What's next

Broaden the reviewed fixture set, connect an explicitly approved live execution path to the same evidence contract, and evaluate the tool with research software engineers. These are planned steps, not capabilities claimed by this demonstration.

Significant updates during this competition

LabRepair builds on an unchanged, attributed MIT-licensed EvidencePilot verification core and earlier Q4 fixture work. That earlier material is included with its license and source pin; the original EvidencePilot repositories were not modified.

Work for this entry adds the Nebius/NVIDIA integration, bounded parameter-to-source repair, synthetic research fixture and counterexamples, recorded model-to-Sandbox validation, interactive review and evidence download, portable judge build, direct Token Factory CLI, persistent daily reservations, and the demo film. No unpublished research, patent materials, or real quantum hardware measurements are used. The clean submission snapshot retains the reviewed current source and attribution without copying the original development Git/PR history or old Release attachments. The original development repository remains private.

Scope statement

This is a bounded, human-authorized, single-pass developer tool. It is not an autonomous repair loop or a general reward-hacking detector. The inputs are synthetic. The original scientific claim remains NOT_REPRODUCED; no physical Bell experiment, quantum advantage or autonomous discovery is established. The browser's incomplete-evidence sample was not part of the recorded three-case Sandbox run. No Tavily integration or OpenShell execution is claimed.

Built With

Share this project:

Updates

Submission history