PatchProof

Inspiration

A plausible patch is not the same as a checked repair. Maintainers need to see which requirements were exercised, what failed, and what evidence supports accepting a change. PatchProof brings those questions into the review workflow.

What it does

PatchProof presents a Python maintenance task with its starting source, explicit requirements, and input/output examples. NVIDIA Nemotron proposes a patch through Nebius Token Factory. Independently controlled checks execute the candidate in isolated local Docker containers, and the interface connects their observations to the requirements.

A reviewer can inspect the contextual diff, check results, unsuccessful attempts, and a downloadable evidence bundle. A human decides whether to use the change. Nothing is merged automatically.

The hosted experience currently offers two original synthetic tasks: configuration overlays and revision pagination. The repository includes a broader ten-task synthetic maintenance corpus. These are curated development examples, not a benchmark of production repositories.

How we built it

The TypeScript/Node.js application combines a model adapter, durable coordinator, SQLite journal, container-based checker, and evidence export. The hosted model is nvidia/nemotron-3-super-120b-a12b through Nebius Token Factory. Earlier recorded attempts also used NVIDIA Nemotron Nano.

Expected outcomes remain under independent host control. Candidate code runs in local Docker containers with networking disabled and bounded resources. Token Factory supplies inference; these containers are not Nebius-hosted Sandboxes. The coordinator limits model calls, tokens, time, and reserved spending. Evidence records the source and runtime, patch, requirement observations, request identity, and reported usage. Token-based estimates are distinguished from reconciled provider billing.

The public app provides live, bounded repair generation and saved results without a sign-in. Its controls can refuse new work when pricing, budget, or execution limits are not valid. A generated repair can fail; the interface retains that result rather than calling partial success a verified repair.

Challenges

The workflow has to preserve useful failure information, bind historical evidence to the exact source and runtime that produced it, distinguish recorded examples from current execution, and fail closed when spending controls cannot be verified. Host load and verification timeouts were practical challenges during development, alongside proposals that did not satisfy every requirement.

Accomplishments

The recorded pagination example started at 16 of 22 declared checks. A real Nemotron Super proposal passed all 22, and a fresh same-host replay reproduced the result without another model call.

The historical configuration example retained a proposal that passed 17 of 22 checks and a later verification timeout. It remained a failed repair. The video shows both recorded outcomes and the evidence-download workflow. It shows September 28 development examples, not a newly executed October run.

The restored hosted service also exposes current results, including a successful configuration repair and a rejected pagination proposal. These examples demonstrate reviewable outcomes, not guaranteed correctness or general repair accuracy.

What we learned

Reviewers need more than a plausible diff or a single success indicator. Requirements, actual observations, and retained failures make a proposed repair's limits inspectable. Model usage and estimated cost also need to remain distinct from authoritative billing.

What's next

Broader maintenance tasks, independent execution environments, and a scored comparison are the next evidence to establish. A human-review study would be needed before making claims about time saved or improved repair rates.

Development and attribution

Documented work during the competition period includes the maintenance coordinator and evidence format, real Token Factory/NVIDIA Nemotron inference, independently controlled container checks, synthetic maintenance fixtures, successful and failed examples, contextual diffs, replay evidence, and the narrated demonstration. October work added licensing, documentation, and restored bounded live generation.

The application and original synthetic fixtures were developed with AI assistance. The demo uses disclosed synthetic narration. Original application code is MIT-licensed; original synthetic fixtures retain CC0 terms, and third-party components retain their own notices.

Try it

Open PatchProof, inspect the Fixed repair and Failed repair walkthroughs, then choose Try a repair. Read a task's starting code, requirements, and examples before choosing Generate patch. Review the actual result and download its evidence. Current runs are bounded to up to 600 seconds; failure or rejection is a valid outcome.

Source and setup · Recorded evidence and limitations · Same-host replay receipt

Built With

• docker
• javascript
• nebius
• nemotron
• node.js
• nvidia
• playwright
• python
• sqlite
• typescript