Reproof

A live triage of a jq report, replayed as RECORDED: the vulnerable build crashed, the fixed one ran clean, tag jq-1.7.1 contains the fix.
100 public bugs picked by a rule set before any ran: 55 crashed on the vulnerable build and 55 ran clean on the fixed one.
All 100 tasks are listed, misses first, each with its reason. Every task that ran links to the card the run wrote.
Pick one of 10 public ARVO reports and press Triage, or show the recorded result, which makes no calls. No login and no key.
On a phone the answer comes first: the recorded jq result with its time, cost and Tavily credits, and a button to run it live.
Both builds run in Nebius sandboxes and code sets the verdict. Nemotron reads the claim and plans searches. Only git ancestry can say fixed.

Inspiration

On May 26 this year, Daniel Stenberg, who leads curl, wrote that "on average we now get more than one report per day." That rate is 4 to 5 times what curl saw in 2024. The reports aren't the old AI slop either. He says they're detailed and long, and he spends almost all of his days working through them: verify the claim, assess it, write the patch, write the advisory. (Source: https://daniel.haxx.se/blog/2026/05/26/the-pressure/)

So the first step of every one of those reports is the same. Does this input actually crash the code, does the fix actually fix it, and has someone already reported it? That's the part I wanted to take off a maintainer's plate. Not the decision. Just the checking.

What it does

You give Reproof a security report. It hands back a triage card with evidence for each line.

Nemotron 3 Super reads the report and pulls out the claim: the bug class and the functions it says crash.
Reproof runs the supplied input against the vulnerable build and the fixed build at the same time, in two disposable branches of a Nebius Token Factory Sandbox (ConTree).
It parses the sanitizer output and compares the real crash with the report's claim.
It searches the public OSS-Fuzz OSV records for a duplicate, after taking out the task's own record.
It asks the open web whether this crash is already public, with Tavily. That part is below.
It writes the card: a verdict (REPRODUCED, NOT_REPRODUCED, DUPLICATE or NEEDS_INFO), the sanitizer excerpt, both sandbox operation ids, the model request ids, the cost, and the time.

A conclusive verdict needs a clean fixed build. Reproof never files anything, never posts anything, and never changes the upstream project. A human reads the card and makes the call.

Try it. https://reproof-web.vercel.app needs no login and no key. Pick one of the 10 featured public reports, or open the 53 more from the 100-bug run and search them, and press Triage. The answer comes first: a verdict stamp and one sentence, the vulnerable and fixed builds side by side, and the public-status line. When git decided the fix on a live run, a ribbon draws the release tag, the word "contains" and the OSV fix commit, straight from the GitHub compare. You can also do it from a terminal:

curl -N -X POST https://reproof-web.vercel.app/api/triage \
  -H "content-type: application/json" -d '{"arvo_id": 42530604}'

That streams one JSON line per step. I ran it on 2026-10-06 at 06:28 UTC: HTTP 200, 8 lines, verdict REPRODUCED, public status PUBLICLY_KNOWN_FIXED, $0.00163346 total, 9.2 seconds. The vulnerable build crashed with a heap-buffer-overflow WRITE of size 1 in decToString, and the fixed build exited 0.

How I built it

Everything that thinks runs on Nebius Token Factory. Everything that executes runs in Nebius Token Factory Sandboxes.

Nemotron 3 Super (nvidia/nemotron-3-super-120b-a12b) does two jobs, both at temperature 0 with thinking turned off and a strict JSON schema. It extracts the claim from the report, and in the public-status stage it plans search queries and classifies each page that passed the gates as the same bug, a related variant, or unrelated. Nemotron never decides the verdict on its own. Deterministic code compares its claim with the measured crash, and only git can say a bug is fixed.

ConTree sandboxes run the proof of concept. I wanted to import the public ARVO Docker images straight in, and that failed on two platform behaviors. Four imports died with NotImplementedError: chmod: follow_symlinks unavailable on this platform, and a wasm3 import came back SUCCESS but couldn't start /bin/bash because a loader symlink was dropped. Both are logged with operation ids in the README and in my feedback. So I built runtime slices instead: Reproof copies just the wrapper, the input, the fuzz target, the symbolizer, the loader and every glibc dependency out of a stopped container, patches the ELF interpreter to /arvo/ld.so, and hashes every file. Each slice goes into a named checkpoint once, then every triage forks two disposable branches from it.

The measured runs. I registered a 100-bug evaluation before running it: the 10 ARVO tasks I'd been using, plus 90 drawn at random (seed 20261019) from 927 ARVO memory-safety crashes that fit my written rules, each with a public OSV record. A task that fails to run still counts against the 100. https://reproof-web.vercel.app/proof lists all 100, misses first, each linked to the card the run wrote.

Run 1 lost its last 49 tasks in six minutes when the connection to Token Factory dropped, because my one retry fired seconds later into the same failure. One more task died on a crash dump with a byte that wasn't UTF-8. I fixed both, then committed a plan for run 2 before starting it: same 100, publish whatever it shows, no run 3. Run 2:

62 of 100 ran. The other 38 got HTTP 500 from the sandbox API, and they count as failures.
55 crashed on the vulnerable build, and 55 fixed builds ran clean.
42 measured crash states matched the one OSV recorded, and 43 report claims matched the measured crash.
7 came back NEEDS_INFO instead of a guess. Six of them are 32-bit fuzz targets that my slicer packed with 64-bit libraries, so neither build could start. That one's my bug.
42 tasks ran in both runs, and all 42 got the same verdict both times.
The run cost $0.32, about a third of a cent per bug.

Before that, the first 10 alone: 10 of 10 completed, 6 REPRODUCED and 4 DUPLICATE, all 10 fixed builds clean, 10 of 10 claims matched, 7 of 10 crash states matched OSV, for $0.0092 total. One thing I got wrong first there: my original table counted four tasks as duplicates of themselves, because OSV has a record for each task's own bug. I caught it reviewing the saved cards, and the duplicate search now excludes every record mapped to the task's own OSS-Fuzz issue.

The judge door is a Next.js page with a Python function behind it on Vercel. The browser never sees the Nebius key or project id. Every id, cost and request id sits under the summary in an Evidence section, one disclosure per step, open while the stream runs and closed when the verdict lands. It's capped at 1 triage at a time per browser, 8 per hour per address and 60 per day overall, and "Show recorded result" replays the stored eval run without calling anything, labelled RECORDED. A second page, /proof, lists all 100 eval tasks and both patch runs from the committed result files.

CI runs Ruff, mypy and 345 non-live Python tests, plus a web job with typecheck, lint, unit tests, the production build and Playwright checks with axe at four screen widths. A daily workflow runs one real triage against the live URL. Two of its first three scheduled runs failed without saying why, so now a failure records every request, status and browser error, saves a trace, and retries once.

Related work. OSS-Fuzz and ClusterFuzz already reproduce the crashes their own fuzzers find. The public ARVO dataset packages reproducible OSS-Fuzz bugs as vulnerable and fixed builds, and Reproof's eval runs on it. What Reproof adds is the other direction: it starts from a report someone sent you, checks the claim against a measured crash on both builds, and then checks the public record with git standing behind any "fixed" claim.

How Reproof uses Tavily

After Reproof reproduces a crash, a maintainer's next question is "is this already public, and is it already fixed?" OSS-Fuzz OSV can't answer it. You can't search OSV by crash text, GitHub's global advisory route returned 404 for all four of the jq advisories I tried, and NVD keyword search by the top frame added nothing in my baseline. The public record of a fresh crash usually lives in an issue thread, a commit, or a project's own advisory page. That's what Tavily search is for.

Here is the stage, every time a triage runs:

Nemotron plans 2 or 3 short queries from the project name, the crash type and the measured frames. A guard drops any query without the project name or a measured frame.
Tavily search (advanced depth) runs at most 2 of them, limited to github.com, nvd.nist.gov, cve.org and the project's own site.
Tavily extract reads the kept pages.
A page only counts if its text contains the top measured frame and the crash type, or at least two measured frames, as exact substrings, and if it lives on the project's own repository or an NVD, cve.org or oss-security page that names the project.
Nemotron classifies what's left, and any quote it gives has to appear word for word on the page (line breaks and repeated spaces aside). If a quote isn't on the page, Nemotron gets one more try, and the second answer faces the same check.
When a kept page is the project's own GitHub advisory, Reproof reads its patched version from GitHub's API. That version is still only a claim.
A "fixed in" claim has to pass git ancestry: GitHub compare must show the stated release contains the OSV fix commit.

It runs in production. The live triage of ARVO 42530604 on 2026-10-06 at 06:28 UTC sent two Tavily searches and one extract and spent 5 credits. Request ids:

2f8216d4-b08e-42ab-a56b-89cbba228098
6c7a310d-91fd-4ad1-b701-1c470e8aaca3
d2c783b6-cffd-4183-a66b-5b7c319dbe32

It kept jq's advisory GHSA-686w-5m7m-54vc (CVE-2023-50246, "heap-buffer-overflow exists in the function decToString in decNumber.c"), which matched 2 of the 3 measured frames. Reproof read that advisory's patched version from GitHub's API, 1.7.1, and GitHub compare showed tag jq-1.7.1 is identical to OSV's fix commit 71c2ab5. So the card says, word for word: "Fixed in jq 1.7.1. Git ancestry: tag jq-1.7.1 contains OSV fix 71c2ab5." A triage of the same crash in the next 60 minutes reuses that lookup, shows those same request ids, and spends 0 new credits.

Why ancestry matters. ARVO 42531223 is a stack-buffer-overflow in jq's decNaNs. When I was researching this stage, a search for it turned up GHSA-7hmr-442f-qc8j, a decNaNs overflow patched in 1.7.1. But OSV's fix commit for this crash, b86ff49, is 102 commits after the jq-1.7.1 tag (GitHub compare: ahead_by 102, behind_by 0). So 1.7.1 doesn't contain this fix, and under Reproof's rule a page saying "patched in 1.7.1" is a related variant, never the fix. In the measured run the advisory Reproof kept for this crash was GHSA-x6c3-qv5r-7q22, the later one.

The measured comparison. I wrote down the rules, the sets and an expected result in the README before any Tavily call, then ran 40 tasks: the 10 reproduced ARVO tasks, plus 30 OSV records drawn with a fixed seed from the 15 OSS-Fuzz projects with the most records.

With Tavily: verified public references on 3 of 10 ARVO tasks and 2 of 30 OSV records.
Without Tavily (GitHub's per-repository advisory API plus NVD keyword search, same tasks): 3 references, all jq advisories, and 0 of 30 on the second set.
Only Tavily found wasm3 issue #458 (the same three use-after-free frames), nDPI commit 412ca87, and PHP issues #18844 and #14969.
The gates threw out a lot before anything reached a card: on the 30-record set, 33 pages for missing frames, 49 for the wrong host, 65 search snippets, 39 source-code files and 2 quotes that weren't on the page.
174 Tavily credits for all 40 tasks (80 searches, 22 extracts), about 4 to 5 per triage.

And an honest miss: in that stored run, Tavily kept no page for ARVO 42530604, while the baseline found GHSA-686w-5m7m-54vc. The live runs on 2026-10-06 did find it, but Nemotron plans the queries fresh each time, so the searches differ run to run. That's why the card always lists the exact queries it sent.

Spend controls. Each lookup reserves 5 credits against 100 per UTC day on that server instance, and sends nothing if the reservation doesn't fit. A call that was sent but got no readable answer still counts against the lookup. If no page could be read, the card says LOOKUP_UNAVAILABLE and claims nothing either way.

Challenges I ran into

The ConTree import failures above. Getting a slice that runs the same in a sandbox as in the original image took patching the ELF interpreter and hashing every file.
Search results move. The same crash got different queries and different pages on different runs. I stopped treating one run as the truth and started recording the exact queries and request ids on every card.
Telling "nothing found" apart from "couldn't look." My first version labelled a failed search as "no public findings", which is a real claim about the world. Now a failed lookup is its own state.
Vercel functions can only write under /tmp, so the first deploy answered every triage with HTTP 500 (Read-only file system: '.cache').
My own live card. On the morning of Oct 6 the jq card kept saying "fix not verified" while the same lookup on my machine said fixed. Running it live and reading the output found three causes. The quote guard compared raw text, so a correct two-line quote failed over line breaks. A quote that missed got no second try. And the door never handed the lookup its GitHub client, so git ancestry had never run on the live site at all. My eval script passed that client, which is why the measured numbers were right and the live card wasn't. Each fix has a test that fails on the old code.
Right after a deploy, Vercel's runtime logs showed my test request answered by the previous deployment. I check the deployment id in the logs now before I trust a post-deploy check.
Getting Nemotron to write a patch that a build proves. My first try failed: all 3 unified diffs Nemotron 3 Super wrote for jq's decToString overflow were rejected by git apply. So I moved patching offline and changed the format. Nemotron now writes exact-text search and replace edits, and the full ARVO image applies them, rebuilds the project, reruns the proof of concept and the seed corpus, with up to 3 tries per bug. I wrote the rules down and pushed them before each run. Run 1 proved 2 of 10 bugs, but I then found half the build scripts couldn't build twice in one container, so 12 of 29 attempts never tested an edit. I fixed the harness to start every attempt from a snapshot, registered run 2, and it also proved 2 of 10: three different bugs across the two runs, one of them in both. The two runs cost $0.073 and $0.091 in tokens. A proven patch isn't the upstream fix, and the live door still doesn't write patches, because a runtime slice has no compiler or source tree.
Running 100 bugs instead of 10 broke things the 10 never touched: a dropped model connection, a crash dump that wasn't valid UTF-8, sandbox API 500s, and 32-bit targets my slicer had never seen. The hard part was keeping the rule I wrote before run 1, that every failure stays in the count, when a third run would have looked better.

Accomplishments that I'm proud of

A 100-bug evaluation picked before it ran, with every failure left in the count and on a public page. Where both runs reached a bug, they agreed on all 42.
A public-status stage where a web page can't make the card say "fixed". Only git can.
A patch loop where a rebuild and the crash itself decide, not the model: Nemotron proposes the edit, the real project rebuilds, and the proof of concept has to stop crashing.
Every number on the page comes from a committed result file or a request id someone else can look up.

What I learned

A model reading a web page will happily say "fixed in 1.7.1". Git is the only thing I trust for that.
A SUCCESS status from a platform isn't proof something works. The wasm3 import said SUCCESS and couldn't start a shell.
Writing the eval rules down before running it kept me honest. The prediction was about 3 of 10 ARVO tasks with a public reference. It came back exactly 3, and the rules were already committed, so there was nothing to tune after the fact.
Tests that inject every dependency can't see one that production forgets to pass. The door test now checks the real wiring.
A retry that fires a few seconds after a dropped connection gets spent inside the same outage. Mine now waits until the endpoint answers again.
My own second-model review rounds caught real bugs in my fixes, like the failed-search label. They also kept going long after they stopped finding code bugs, so I set a stop rule.

What's next

No maintainer has run Reproof on a real incoming report yet. That's the test I want next: one maintainer, one real report, and their call on whether the card saved them time.

The patch eval proved 2 of 10 in each run, and temperature 0 still gave different replies between runs. Four of run 2's 28 replies hit the 4,096-token output cap, so a bigger cap is the first thing I'd register for a next run.

Right now the door takes 63 ARVO reports: the 10 featured ones and 53 from the 100-bug run whose sandbox checkpoints were built and measured. A new report means building its slice first. Next on the slicer: read a 32-bit target's libraries from the 32-bit directories. That mismatch is behind six of run 2's seven NEEDS_INFO cards.

Live demo: https://reproof-web.vercel.app

Every bug in the 100-bug run: https://reproof-web.vercel.app/proof

Code: https://github.com/StephenSook/reproof (Apache-2.0)