ShipCheck dashboard: a real scan of the vulnshop demo app. 4 of 4 planted vulnerabilities exploited, fixed and re-verified under a minute
Confirmed SQL injection: payload admin'--, marker [email protected], Tavily CWE/CVE grounding, and the parameterized-query fix.
The draft PR ShipCheck opened: exploit evidence, advisories, diff and re-verification. It never auto-merges; a human decides.
A real scan in 54s: semgrep/gitleaks/osv triage, Nemotron Ultra plans, live exploits in sandboxes, Nemotron Super fixes.
Every push runs the full suite on GitHub Actions: 14 consecutive green runs on main.
CI on Python 3.11 and 3.12: 135 tests plus live exploits fired against the demo app on every push.

Inspiration

AI assistants now write a large share of production code, and that code passes tests. The tools meant to catch its security bugs only ever suspect: scanners and LLM reviewers produce findings that are often wrong, so teams learn to scroll past them. We wanted a gate that never asks you to take its word for it. If ShipCheck says your PR is exploitable, it shows you the exploit working. If it says the fix is good, it shows you the same exploit failing.

What it does

For every pull request, ShipCheck runs a five-stage loop:

Triage - semgrep (OWASP Top 10, plus a path-traversal taint rule we wrote because no registry pack caught ours), gitleaks, osv-scanner and pip-audit flag suspects in the diff. Overlapping rules on the same sink are merged into one finding.
Plan - NVIDIA Nemotron 3 Ultra on Nebius Token Factory reads the diff, the triage findings and the full source of every affected file, and writes one falsifiable attack plan per finding: route, HTTP method, parameter, payload, and a deterministic success marker. Tavily supplies current exploit write-ups.
Attack - each plan fires as a real HTTP request against the running app inside its own forked, disposable sandbox. A finding is confirmed only when the marker appears in the response: the admin's email for SQL injection, uid= for command injection, root: for path traversal. No LLM casts a vote. Confirmed findings get a deterministic 0–100 risk score (severity × confirmed × reachability × exposure) and Tavily CVE/CWE grounding with the canonical fix.
Fix - Nemotron 3 Super writes a patch from the exploit evidence. ShipCheck applies it in a forked sandbox, re-runs the exact same attack (the marker must be gone) and runs the repo's own tests (they must pass). If either check fails, Super gets the failure output and tries again. No PASS, No PR.
Hand-off - A draft pull request containing the exploit evidence, the advisories, the root cause, the diff, the re-verification result and a reviewer checklist. ShipCheck never merges. A human decides.

On our vulnshop demo app, a real run confirms 4 of 4 planted vulnerabilities, fixes all four, re-verifies them and opens a real draft PR — in under a minute.

How we built it

Python 3.11, Pydantic v2 contracts frozen in week one so three people could build in parallel, FastAPI and a dependency-free dashboard.
Nebius Token Factory through its OpenAI-compatible API: nvidia/Nemotron-3-Ultra-550b-a55b plans the attacks, nvidia/nemotron-3-super-120b-a12b writes the fixes.
Tavily for exploit research and per-finding advisory grounding.
Sandboxes behind one interface, three implementations: a Nebius Token Factory Sandboxes (ConTree) backend, a Docker backend (one container per sandbox, forked with docker commit) and a local temp-dir backend.
GitHub REST API for draft PRs. A re-scan refreshes the same PR with a fast-forward commit and never touches a branch a human has pushed to.
A golden-run replay, so the public demo never depends on a live call: a recorded real scan, redacted, served from the dashboard.
135 tests, plus live exploit tests and Docker tests. CI runs on Python 3.11 and 3.12 and includes live exploits against the demo app.

Challenges we ran into

Nebius Sandboxes access. Our key worked for inference, but every Sandboxes call returned 403; /v1/whoami showed every permission as false, because the beta had not been enabled for our project. We built a Docker backend with the same probe image and the same fork semantics, so the pipeline still runs fully isolated. The recorded demo run used Docker sandboxes, not Nebius Sandboxes.
Making model output safe to act on. Ultra followed our first prompt's outdated schema exactly, so every plan failed validation and we silently fell back to templates. Super sometimes appended a stray character to a JSON string. Given only a diff, the planner guessed routes and invented markers. The fixes: canonical-schema prompts with per-plan validation, feeding the planner whole files, exact-match patching with a narrow repair, and feeding gate failures back to the model.
Our own re-attack hit unpatched code. The first real run "fixed" nothing, because the re-attack ran against the parent sandbox instead of the patched fork. A verification gate that cannot verify itself is worthless, and this is exactly the class of bug it exists to catch.

Accomplishments that we're proud of

An honest gate. Every confirmed finding has a real exploit in the transcript, and every accepted fix has a real failed re-attack plus green tests behind it.
Real end to end: real scanners, real models, real HTTP attacks, a real draft PR opened on our repo.
It caught our own bugs. The verification loop found three defects in ShipCheck itself during development.

What we learned

Deterministic markers beat model judgement for deciding "is this exploitable". A string match cannot hallucinate.
Give the planner the whole file. Long context is not a luxury for security planning; with only a diff, the model guesses.
Feed failures back into the loop. The verification gate turned out to be the best teacher the fixer model had.

What's next for ShipCheck

Run on Nebius Sandboxes as soon as access is granted, and re-record the demo there.
More vulnerability classes: SSRF, SSTI, IDOR and deserialization, which already have templates.
Semgrep taint-based reachability instead of the current heuristic.
A GitHub App that runs on every pull request.