Exhibit A

Inspiration

So much code now gets written by AI and shipped by people who can't read all of it, and it ships with the same handful of security holes every time. The scanners meant to catch that throw hundreds of warnings, most of them false alarms, so people stop trusting any of them. I wanted to try the opposite: a tool that has to prove a bug is real before it's allowed to tell you about it.

What it does

Exhibit A scans a repo and, for each suspected vulnerability, builds a case:

1. It writes a unit test that asserts the secure behavior — a test that fails on the current, vulnerable code. That failing test is the evidence.
2. It runs that test in an isolated sandbox to confirm it actually fails.
3. It writes the smallest patch it can, then re-runs the proof and the project's existing tests until the proof passes and nothing else broke.
4. It hands back a diff and a short case file ready for a pull request.

If it can't produce a test that reliably fails, it drops the finding instead of reporting it. So you get no unproven findings, and every fix is checked by running real code.

How I built it

• NVIDIA Nemotron on Nebius Token Factory, routed by task: Nano triages every file by risk, Super turns the risky files into concrete findings, and Ultra does the hard reasoning — writing the proof test and the patch.
• A sandbox runs pytest against a copy of the target repo: Docker with no network and dropped capabilities, or a locked-down local subprocess as a fallback.
• FastAPI + Server-Sent Events stream the agent's progress to a case-file UI.
• A bundled intentionally-vulnerable Flask app (NoteVault) to test against safely.

Challenges I ran into

Getting a model to write a test that fails for the right reason — a real security assertion, not an import error — took careful prompting and a retry loop that feeds the pytest output back to the model. Making sure a patch didn't quietly break the rest of the app meant re-running the whole test suite after every fix and reverting if anything regressed.

Accomplishments that I'm proud of

Every finding it reports is backed by a test that actually ran. On a real run against an OWASP vulnerable Flask app it had never seen, it proved and fixed a real SQL injection and a template injection, found a bug I hadn't planted, and refused to report the ones it couldn't prove — all for about 13 cents.

What I learned

Grounding the agent in real execution — making it run a test instead of just asserting things — is what makes it trustworthy. And routing cheap models for breadth with a big model held back for the hard steps keeps the cost sane without losing quality.

What's next for Exhibit A

More languages (JS/TS, Go), auto-opening the pull request, and a GitHub Action that runs Exhibit A on every PR so only proven issues ever reach a human.

Built With

• docker
• fastapi
• flask
• llm
• nebius
• nvidia-nemotron
• pytest
• python