MUTINY: Don't trust the PR title. Run the code.

AI writes code faster than anyone can review it. MUTINY runs the old and new code of a pull request on Nemotron-written inputs in Nebius Sandboxes, and hands you the input where they disagree.

Comment

Inspiration

A pull request in Google's GenAI SDK was titled "chore: expose GenAI domain types in dedicated submodules". The tests passed, it was merged and released, and it broke genai.voices, which raises AttributeError from 2.26.0 through the latest 2.33.0.

That is the failure modern review is built to miss. Published work finds that 19 to 35% of LLM-generated refactorings are functionally incorrect, and over 21% of those still pass the project's tests. Agents now write code faster than anyone can review it, and every AI review bot answers "did this change behaviour?" the same way: it reads the diff and gives an opinion. We wanted the observation instead.

What it does

MUTINY takes a pull request, fetches the code at the merge base and at the head, and runs both versions on the same generated inputs. Where they disagree, you get the witness: the exact call, what it returned before, and what it returns now.

  • Nemotron 3 Super, on Nebius Token Factory, writes inputs, never assertions. A wrong input is rejected by both versions and costs nothing; a wrong assertion would manufacture a false finding. Only execution produces evidence.
  • Nebius Sandboxes run both versions. The code is installed once into a warm checkpoint, then forked: inputs are spread across microVMs, 40 forks in 3.7 seconds. A pull request that hangs, segfaults or deletes files costs nothing, and untrusted code never runs on the server.
  • It knows what the pull request claimed. Nemotron compares the difference with the PR's own title and description. A change the PR announces is reported as doing what it says; one it does not mention is flagged.
  • Tavily checks whether the old behaviour was promised. It searches the project's documentation, keeps only the project's own pages and extracts them; Nemotron must quote one sentence word for word, and the quote is shown only if it really is on the page.
  • It ends with what to do: a review comment carrying the proof, and a mutiny pin command that turns the witness into a test.

It runs where code is written: a GitHub Action that comments only on undescribed changes, an MCP server a coding agent calls on its own edits, a CLI, and a hosted demo at https://mutiny-verify.vercel.app with a guided tour.

Does it work? The census

We pointed it at 266 merged pull requests from the 300 most-downloaded PyPI packages, every one claiming it changed nothing, and froze that list in git before running anything on it.

  • 3 regressions. A werkzeug rename that would have broken Flask, reverted by the maintainer five days later ("still private, but flask uses it"). IPython reporting real WebP images as PNG in four releases, fixed 106 days after merge. And the google-genai voices break, still in the latest release.
  • No AI review bot was on any of the three. Bots reviewed 42 of the 266. We ran their claims too: two were real breakages that shipped (isort, pyjwt) on code MUTINY could not reach. Reading and running catch different things.
  • 0 of 224 placebo pull requests flagged. These are PRs whose code could not change behaviour.
  • 8 of its 27 flags were its own mistake, every one read by hand and classified with the fix it argues for.
  • $4.15 for all 490 runs, under a cent each.

How we built it

  • Python engine (mutiny/) that yields one stream of events, rendered by the CLI, the web page (FastAPI and server-sent events, on Vercel) and the GitHub Action alike.
  • NVIDIA Nemotron 3 Super 120B on Nebius Token Factory for every model job: writing inputs, building objects that are hard to construct, rewriting a function when there is no PR, summarising a witness (reasoning off), judging the PR's intent (reasoning on), and quoting documentation. Nemotron 3 Ultra is the fallback. The client adds a spend cap, a ledger, response caching and reasoning-aware retries.
  • Nebius Sandboxes (the ConTree SDK) for the warm checkpoint and concurrent forks.
  • Tavily search and extract for the documentation check.
  • An agent loop for hard objects. When no generated input can build the receiver, the agent reads the class, writes a recipe, runs it, and retries until it works, before any probe is written.

Choosing models by measurement

  • On an identical prompt for 45 inputs, Super returned 42 usable inputs in 10.7 s with no reasoning; Nano deliberated for 26,300 characters; Lightning produced 1.
  • Reasoning is off for summaries (about 6x fewer tokens, at least as accurate) and on for judging intent, because turning it off there flipped 7 of 12 judgements to "not described".
  • Across the census, about 3,000 Nemotron calls cost $4.15, against $35.72 for the same calls on Ultra.

Challenges we ran into

  • Nondeterminism makes false findings. Memory addresses, set ordering, hash seeds, random and os.urandom each produced confident, specific, wrong results. Now every divergence must repeat on the same version before it is reported.
  • The worst bug was a silence. A "repaired" rewrite hung on every input, every observation was lost, and the run said "preserved". Each probe now has its own timeout, and nothing is called preserved unless every input was observed on both sides.
  • Being honest at scale. The census showed the tool's own failure modes: inputs that break a type's contract, probe setup that names a removed global, and an intent judgement that sometimes misses a description written in other words. All are written up rather than hidden.

Accomplishments that we're proud of

  • A pre-registered census on real, popular code, with three regressions reproduced by hand, one of them independently reverted by its maintainer.
  • Zero false alarms on 224 placebo pull requests, and every mistake on the rest classified in public.
  • A head-to-head with the AI review bots projects actually install, where every claim was settled by running it.

What we learned

The model is most useful when it is never allowed to be the evidence. Bigger was not better: the shape of the task decided the model. And a verification tool that cries wolf gets switched off in a week, so false alarms deserve more engineering than detection.

What's next

Fix the three false-positive causes the census found (check generated stand-ins against the real type, discard probes whose setup fails, ignore identity-derived values), have a private helper's divergence trigger a check of its public callers, and widen coverage: no input could be built for 98 of 253 census pull requests.

Built With

  • contree
  • driver.js
  • fastapi
  • github-actions
  • javascript
  • model-context-protocol
  • nebius-sandboxes
  • nebius-token-factory
  • nemotron-3
  • nvidia-nemotron
  • python
  • server-sent-events
  • tavily
  • upstash-redis
  • vercel
Share this project:

Updates

Submission history