Inspiration
AI tools can now suggest fixes for security bugs, but a suggested fix is only a guess. It can pass a project's tests and still leave the hole open. We wanted to measure how often that happens, and to see whether an AI agent could catch its own mistakes. [Add one or two sentences in your own words about what made you care, such as a moment you saw an AI fix that was wrong.]
What it does
PatchProof runs AI-written security patches inside a locked-down Docker sandbox and judges them with a verifier written as plain code. In the benchmark it also uses a hand-written hidden oracle test that the model never sees. A patch counts as verified only if the existing tests pass and the hidden oracle passes.
We used it to measure four NVIDIA Nemotron models (Lightning, Nano, Super and Ultra), served through Nebius Token Factory, on eleven hand-built vulnerabilities: SQL injection, path traversal, prefix path traversal, ORDER BY injection, command injection, unsafe eval, HTML escaping, open redirect, LIKE-search injection, zip slip and header injection. We ran over a thousand patch attempts.
What we found
- One-shot fixes pass easy cases and fail subtle ones. On five common flaw types, 99 of 100 one-shot patches were verified. On open redirect, all 20 passed the ordinary tests and failed the hidden oracle, because a form like ///evil.test/ still slipped through.
- A hidden oracle catches what ordinary tests miss. Across the eleven cases it flagged 35 one-shot patches that looked correct and were not.
- An AI-written attack test with retries did not help. Across four models and eight cases, 112 of 160 patches were verified with the loop against 126 of 160 one-shot, at about 12 times the tokens.
- A longer attack list made it worse. On three held-out cases built after the library was written, a general attack-variant library verified 40 of 80 on two cases, against 67 of 80 for the plain loop and 76 of 80 one-shot. Lightning was truncated on every run with the library.
- A hybrid that keeps the one-shot patch and uses the AI test only as extra evidence tied the one-shot fix, 183 of 220 against 183 of 220, at about 11 times the tokens. Its confidence label did not predict correctness, so we dropped it.
- The reason we found: the AI-written tests cover attacks the model already knows. On real code there is no hidden oracle, so a verdict is only as strong as the AI-written test.
How we built it
Python agents call the Nemotron models through the Nebius Token Factory OpenAI-compatible API. Every test runs in Docker with no network, a read-only filesystem, dropped capabilities, a non-root user, resource limits and a timeout. The verifier hash-checks the test files so a patch cannot edit the tests it is judged by. A Tavily web search adds advisory context to the test-writer in one agent variant. The results site is generated from the raw result files and hosted on GitHub Pages.
Challenges we ran into
- Reasoning models spend output tokens on thinking, so our first runs looked like model failures but were truncated replies. We added a separate truncated outcome.
- At the API's default temperature, one model returned identical replies, so some early repeats were not independent. The main evaluation used temperature 0.7.
- One model disappeared from the account's list partway through, and a network drop corrupted a batch. We labelled both as infrastructure errors and excluded them from the rates.
- Two of our own oracles had holes, one checking an issue the finding did not mention and one with a parser that missed bare line breaks. Validating each oracle against a known-good and a known-naive fix is what caught them.
- Asking a model to red-team its own patch produced bypasses that were really the allowed host, so we dropped that idea.
Accomplishments that we're proud of
- A sandbox verifier we trust, validated on every case with a known-good and a known-naive fix.
- Honest measurements: repeated runs, confidence intervals, a held-out test, and negative results kept in the write-up.
- A concrete result that a hidden oracle exposes false fixes that ordinary tests, and the model's own tests, miss.
What we learned
Verification is only as strong as its test. Models write attack tests from the attacks they already know, so the interesting failures are the ones nobody thought to test. A single call was hard to beat, and every extra step we added cost far more than it gained. Honest evaluation also meant separating infrastructure problems from model failures.
What's next for PatchProof
- Test generation seeded from real parser-difference research, evaluated on fresh held-out cases.
- More vulnerability classes and larger samples.
- Using several independent models to write competing attack tests before a patch is accepted.
Log in or sign up for Devpost to join the conversation.