NVIDIA Nemotron and Nebius
Shadow Patch

Inspiration

Code generation is only one step in fixing a bug. A proposed patch is not proof that the bug is fixed. Maintainers still need to reproduce the problem, verify that the regression test is meaningful, and ensure that the fix does not break existing behavior.

We built Shadow Patch to make this evidence part of the issue-to-PR workflow. Instead of trusting an agent's explanation, Shadow Patch requires a regression test, isolated candidate evaluation, and a clean replay before presenting a repair for human review.

What it does

A maintainer applies the "shadow-fix" label to a GitHub issue. Shadow Patch checks the selected sandbox runtime and existing test baseline before making model calls.

A verifier model generates a regression test using the issue and original repository. Shadow Patch confirms that the test fails on the unfixed code because of an actual assertion failure. The test is then frozen and withheld from the repair models.

Three separate solver strategies propose candidate fixes. Each candidate runs in an isolated Nebius Sandbox and must pass both the existing tests and the frozen regression test. Shadow Patch selects the best passing candidate (fewest changed files, then fewest changed lines, then fastest) and replays it from a clean sandbox image.

After successful verification, the workflow opens or updates a pull request containing the code changes and verification report. If verification fails, the workflow stops without opening a new PR and comments the reason on the issue. A developer always makes the final merge decision.

One Nebius API key covers both Token Factory inference and Sandboxes, so a full run (a verifier plus three solvers) fits inside a single GitHub Actions job.

How we built it

Shadow Patch is orchestrated in Python through GitHub Actions and is packaged as a reusable GitHub Action. It uses NVIDIA Nemotron models through Nebius Token Factory for regression-test generation and candidate repair. Our recorded development runs initially used "nvidia/Nemotron-3_5-Lightning" and later used "nvidia/nemotron-3-super-120b-a12b". We also tried "nvidia/Nemotron-3-Ultra-550b", which used too many tokens for this workload.

Nebius Sandboxes, accessed through "contree-sdk", provide isolated environments for repository setup, baseline tests, regression checks, candidate evaluation, and clean-image replay. Runtime adapters separate repository and toolchain configuration from the verification engine. For Node.js and TypeScript projects, the v0.6 engine uses an explicit, pinned "tsx" test-running contract.

We also implemented safeguards around the verification process:

Test before repair: The regression test is generated and validated against the unfixed code before candidate fixes are attempted. Hidden regression: The regression test is excluded from solver prompts, reducing the opportunity for candidates to tailor their fixes to the test. Isolated evaluation: Each candidate is tested in its own sandbox branch. Source-path restrictions: Designated protected files cannot be edited directly by the repair process. Integrity checks: Hash comparisons help verify that regression-test bytes remain unchanged. Clean replay: The winning patch is reapplied and verified from a fresh sandbox image before pull-request creation.

These controls provide inspectable test evidence. They do not establish formal correctness or guarantee security against hostile code.

Challenges we ran into

Early runs exposed problems with inference-response parsing and repository-specific test setup. We also encountered a candidate that changed existing low-speed formatting. Although the proposed fix looked plausible, verification rejected it because it did not preserve the required behavior.

We improved validation feedback, failure-stage reporting, bounded corrections for baseline failures, and pull-request notifications. We also refined the TypeScript test contract so regression tests could run more consistently in the sandbox.

Later challenges were environmental rather than model-related. A newer jsdom crashed on load under the sandbox's Node 20, so we pinned a compatible version and now check it before any model request. Tests created at the site root could not fail for subpath bugs, so the engine now refuses them. A transient Nebius Sandbox API error ended a run, so setup steps now retry and are reported as infrastructure errors instead of repair failures. Projects whose tests ran only a named file, or whose tsconfig used project references, were blocked until we made them testable.

We learned to treat these failures as engineering feedback rather than evidence that the system works everywhere.

Accomplishments that we're proud of

We built an end-to-end workflow that takes a labelled GitHub issue through regression-test generation, candidate repair, isolated verification, clean replay, and pull-request creation.

In a recorded repair of a TypeScript utility, Shadow Patch evaluated three candidates, rejected a candidate that produced incorrectly formatted output, and selected a passing alternative. Clean replay passed the existing baseline tests and regression tests before the workflow updated the pull request.

We also recorded further repairs involving a cryptographic bug, Wi-Fi QR-code escaping, a security scanner's Axios version check, and static-site deployment behavior. All of these are on repositories we own. They demonstrate progress across several development scenarios.

What we are most proud of is the verification-first design: a repair must demonstrate that the reported bug can be caught by a regression test, preserve existing test behavior, and pass again when replayed from a clean environment.

What we learned

We learned that generating a plausible fix is not necessarily the hardest part. Creating a reliable regression test that runs correctly in a repository's actual environment can be just as challenging.

We also learned that verification must be independent of the repair process. Keeping the new regression hidden from solver prompts reduces the risk of candidates tailoring their implementations to a visible test. Clean replay provides an additional check that the winning patch works outside its original candidate environment.

Our benchmark work taught us to distinguish safe rejection from broad capability. In an earlier benchmark, the engine accepted all six trials on repositories it had been developed against but failed to accept tests in six trials on unseen repositories.

That lesson now shapes our approach: report limitations clearly, fail closed when evidence is insufficient, and expand testing on unfamiliar repositories before making broader claims.

What's next for Shadow Patch

Our next priority is improving regression-test generation on unfamiliar repositories, particularly where nested test layouts, repository-specific module aliases, or API type-checking cause problems.

We plan to:

Evaluate more held-out issues and repositories. Add and improve runtime adapters for different languages and test frameworks. Strengthen the evidence linking each verification result to the exact source revision and test artifact. Improve isolation and defenses against hostile repository code. Add clearer execution-cost reporting and explore parallel candidate evaluation (candidates currently run one after another).

Longer term, we aim to evolve the GitHub Actions implementation into an installable GitHub App, making verification-first repairs easier for maintainers to adopt without sacrificing human review.