ReproPilot

For open source maintainers who triage more bug reports than they can reproduce by hand. ReproPilot turns a report into a failing test and a verified fix. It runs both before it reports anything.

In the recorded demo run the whole loop took 10.1 seconds: three Nemotron calls on Nebius Token Factory and three pytest runs. It ended with a one-line patch that passes the two new tests and all 7 tests in the suite.

Where the submission answers each judging criterion:

• Technological Implementation: How we built it, Challenges, and the code in src/agent/.
• Design: the hosted demo and the video. Every step, model call and test run is visible.
• Potential Impact: What it does, What's next.
• Quality of the Idea: Inspiration. The test run decides, the model only proposes.

Inspiration

Most bug reports arrive without a reproduction. Before anyone can fix the bug, someone must read the report and find the code. Then that person writes a test and checks that it fails for the reason in the report. Language models are good at drafting that test. The trouble is that an unrun test proves nothing. It can pass on the buggy code, or fail with an import error that has nothing to do with the bug. We wanted an agent that only says "reproduced" after it sees the test fail for the right reason.

What it does

You give ReproPilot a GitHub issue URL, or a pasted report and a repository URL. It downloads the code into a fresh workspace, triages the report, picks the relevant files, and writes a reproduction test. Then it runs the test.

• If the test passes on the buggy code, ReproPilot rejects it.
• If the test errors before it reaches an assertion (import error, collection error, missing fixture), ReproPilot rejects it. The real output goes back to the model.
• Only a test that fails on an assertion counts as a reproduction.

After that it asks for a patch and keeps it only if the reproduction test passes and the full test suite still passes. The output is the failing test, a unified diff, all run logs, and a pull request description. The web UI streams every step and every rejected attempt.

How we built it

• Next.js and TypeScript for the UI and an API route that streams NDJSON events.
• NVIDIA Nemotron models on Nebius Token Factory through the OpenAI-compatible API. Nemotron 3.5 Lightning does triage. Nemotron 3.5 Super writes tests and patches. A larger model makes the last patch attempt.
• Two test runners behind one interface: local and Docker. The recorded run used the local runner.
• A failure classifier that separates "fails on an assertion" from "broken before the test ran" for pytest, vitest and jest.
• The hosted demo is a static export of the same UI on a Cloudflare Worker. It replays a run recorded on 2026-10-29 with live Nemotron responses from Token Factory and real pytest runs, because a static host cannot run tests.

Challenges we ran into

• Telling a real reproduction from a broken test. A non-zero exit code is not enough. The classifier reads the test output and rejects collection and import errors.
• Model output format. Not every model on Token Factory supports JSON mode, and some wrap answers in reasoning blocks. ReproPilot asks for JSON mode, retries without it on HTTP 400, and strips <think> blocks and code fences before parsing.
• Reasoning tokens. Nemotron 3.5 spends several hundred reasoning tokens before it answers. Our first live triage call had a 1,024-token limit and came back empty. We raised the limit. If a reply stops with no content, the agent retries it once with twice the budget.
• Model availability. The escalation model ID is not in the public docs. The model list returned it once we had a key. If that model returns a 404, the agent falls back to the Super model and the run continues.

Accomplishments that we're proud of

• The agent cannot claim success without evidence. Every verdict links to the exact command and output that produced it.
• The rejected attempts are visible, which makes it easy to see why the agent changed course.
• The demo recording is a real run, not a mock. The answers came from Nemotron on Token Factory and the test output came from pytest, recorded on 2026-10-29.

What we learned

Running the code changes what you can trust. Once the test result decides, the model only has to be good at proposing candidates, and its mistakes become cheap retries instead of wrong answers.

What's next for ReproPilot

• A GitHub App that comments on new issues with a verified failing test.
• More languages and test frameworks (Go, Rust, Java).
• Running each test in Nebius Token Factory Sandboxes after Nebius approves our beta access request. No test runs there yet.

What breaks first at scale is context selection. The agent reads a handful of source files, and on a large repository it can pick the wrong ones. Better file retrieval comes before anything else on this list.

Built by

JunTae Hahm, entering solo. I work as a frontend developer. We wrote all code in the repository during the hackathon, starting 2026-10-28.

Built With

• cloudflare-workers
• nebius-token-factory
• nextjs
• node.js
• nvidia-nemotron
• openai
• pytest
• python
• react
• typescript