PatchBay

Try it

• Demo (replay of recorded live Nemotron + Tavily runs; tests and patches still execute for real when run locally): https://patchbay-demo.vercel.app
• Code (MIT): https://github.com/LevAdamenko/patchbay — the README runs the full pipeline offline in about two minutes, no API key needed.
• There is intentionally no public live mode: PatchBay executes code from vulnerable library releases, so it only runs in a local sandbox.

Inspiration

Every week brings new CVEs in the open-source libraries everyone depends on, and maintainers are already drowning. AI agents can write patches quickly, but a model saying "fixed" is not evidence — a plausible patch that doesn't actually close the hole is worse than no patch, because it gets merged. I wanted an agent whose output a maintainer can trust without re-doing the work: one that proves the vulnerability, proves the fix, and proves the test would catch a regression.

What it does

Give PatchBay a CVE id and it:

1. Researches the advisory, affected versions, CWE and upstream fix with Tavily, restricted to advisory sources (GitHub Advisory Database, NVD, OSV, Snyk).
2. Has NVIDIA Nemotron 3 Super (on Nebius Token Factory) write a structured brief and a test that reproduces the vulnerability. The test must be red on the vulnerable release — by assertion or timeout, not by import error — and green on the release with the official fix.
3. Has Nemotron write a patch that turns the test green while the library's own upstream regression tests stay green.
4. Runs a deterministic mutation gate: every hunk of the patch is reverted and the added lines are mutated; the test must catch every reversal and at least 80% of operator mutants. Surviving mutants go back to Nemotron for a narrower patch.
5. Renders a pull-request-style report where every claim cites a source or carries a SHA-256 of the test, patch or sandbox log, plus one digest over all evidence. If anything is missing, the renderer refuses.

The demo covers three real CVEs of different classes — a ReDoS in python-multipart (CVE-2024-24762), a proxy-credential leak in requests (CVE-2023-32681) and an attribute injection in Jinja2 (CVE-2024-22195). All three were produced by live Nemotron runs and pass the mutation gate at 100%.

How I built it

• Python pipeline of six stages emitting events; a FastAPI web UI streams them over server-sent events, and a static replay page (served from Vercel) plays back the recorded run for judges.
• Nemotron 3 Super 120B (nvidia/nemotron-3-super-120b-a12b) via Token Factory's OpenAI-compatible API, with Nemotron 3 Ultra 550B as fallback. It is used for the brief, the repro test, the patch and the patch retry after the gate.
• Tavily search (search_depth=basic) grounds the brief in real advisories instead of the model's memory; each brief field links to a retrieved URL.
• A subprocess sandbox runs model-written code: temporary directory, environment stripped of API keys, wall-clock timeout, CPU/memory/process limits.
• Record/replay fixtures: live Nemotron and Tavily responses are recorded with request hashes that mask run-to-run noise, so anyone can replay the full pipeline offline without keys — while the tests and patches still execute for real.
• 141 automated tests, including mutation probes of the gate and the report guard itself.

Challenges I ran into

• Tests that "prove" the bug. Nemotron often wrote tests asserting the vulnerability exists — red on the vulnerable code, but no patch could ever turn them green. The fix was an oracle the model never sees: every repro must also pass on the official upstream fix.
• Unified diffs from an LLM rarely apply. Hunk headers were off by a few lines. I switched to SEARCH/REPLACE edit blocks and compute the diff myself.
• Reasoning budget. As a reasoning model, Nemotron sometimes spent its whole token budget thinking and returned an empty answer; I raised max_tokens to 32k and retry once without thinking.
• Deterministic replay. Sandbox logs fed back to the model contained durations and memory addresses, so recorded request hashes drifted between runs. I mask that noise before hashing.

Accomplishments that I'm proud of

• The pipeline refuses rather than bluffs: no evidence, no report.
• Three different vulnerability classes, all fixed by live Nemotron output, all at 100% mutation score.
• Judges can verify everything offline in about two minutes, without any API key.

What I learned

Agentic coding gets trustworthy when the checks are deterministic and the model can't see them. The interesting engineering is not the prompt — it's the oracle, the gate and the evidence chain around it.

What's next

• A larger curated catalog of CVEs, and multi-file patches.
• Opening real pull requests against forks, with the report as the PR body.
• Running the sandbox as Nebius Serverless Jobs so many CVEs can be processed in parallel.

Safety

Curated catalog only — users can't enter arbitrary CVEs or upload code. Vulnerable releases are pinned, sha256-checked local wheels; nothing runs against remote systems; the "exploit" is a unit test calling a local function.

Built With

• fastapi
• nebius-token-factory
• nvidia-nemotron
• pytest
• python
• tavily
• vercel