Breakwater

A red team for performance PRs. Nemotron finds the inputs a speedup’s benchmark forgot; it caught 9/9 real pip regressions the PRs’ own tests missed, for 18¢ an audit.

Comment

Inspiration

AI-written optimisation PRs are now routine, and a benchmark only tests the inputs someone thought of. This year pypa/packaging, the version parser inside pip, merged a run of performance PRs by careful human maintainers. Several quietly changed behaviour: one accepted "1. 0" and Arabic-Indic digits as versions, one broke SpecifierSet equality, one made old pickles fail to load. Every one passed its tests and its benchmark. We wanted a reviewer that treats the benchmark as the suspect.

What it does

Breakwater is a red team for performance pull requests. Give it a repo with base and head refs, or add the GitHub Action, and a Nemotron adversary reads the diff, works out which assumptions the fast path relies on, and writes attacks to break them. Every line of code the models write runs in a Nebius Token Factory Sandboxes microVM, never on your machine. You get:

  • a verdict: BLOCK, REVIEW or PASS;
  • minimal counterexamples, each as a ready-to-commit pytest verified to pass on base and fail on head;
  • per-family slowdown measurements from deterministic instruction counts;
  • a bill by model tier, with the sandbox image UUID behind every number.

What it found:

  • pip's version parser: all 9 documented regressions across 29 packaging speedup changes, caught from the parent commit with no hint of which bug exists. The PRs' own tests caught 0; the best no-LLM baseline caught 7. A re-run on Sandboxes caught all 9 again.
  • Merged AI-written speedups: 4 of 10 Codeflash-bot PRs change behaviour, 3 adjudicated likely-bug. One, "Speed up ObjectComparer.is_same_object by 109%", does up to 363× the instructions on two same-sized JSON objects that differ by one key. Plain Hypothesis and mutation testing found nothing in these repos.

How we built it

  • Nemotron 3 Ultra (Token Factory, reasoning on, medium_effort) reads the diff white-box: it traces the old code, then plans attacks that each predict a base/head difference. It also plans slowdown input families and adjudicates every finding as likely-bug, intended or needs-human, three votes each. A vote counts only if the evidence it quotes appears verbatim in the PR.
  • Nemotron 3.5 Lightning (reasoning off) writes the high-volume part: Hypothesis strategies and input generators, about twenty per audit.
  • Nemotron 3 Super writes variants of the slowest perf families.
  • Token Factory Sandboxes runs all of it. Base and head are installed into one kept image; every strategy, probe and measurement executes in a microVM branched from it with networking removed (358 runs in the demo audit). Deterministic probes compare outputs, exception types, dunders and cross-version pickles outside both processes.
  • Performance is measured with cachegrind instruction counts rather than time. For each input family and size $n$ we report $r(n) = \mathrm{Ir}{\text{head}}(n) \,/\, \mathrm{Ir}{\text{base}}(n)$, and a family whose $r(n)$ grows with $n$ is flagged as a complexity regression.
  • A static report site with an in-browser judge box running four real versions of packaging (Pyodide), a live audit page with replay, and a GitHub Action that posts one PR comment.

Challenges we ran into

  • Getting Ultra to plan attacks that predict a difference instead of generic fuzzing ideas. Planner v3, which must trace the old code before proposing anything, took productive attacks from 0/15 to 7/15 on a plan-only eval.
  • Structured output that was schema-valid but empty, or that ran into the token cap. Pinning array bounds and capping each call at 8 items fixed it.
  • Measuring speed without noise. CPU time varied by about 10% between identical runs, even inside a microVM. Instruction counts are deterministic on a given image, so that is what we publish.
  • Honest provenance. The same family measured 4.5× on arm64 Docker and 6.5× on the x86_64 microVM, so every number now names its backend and image.

Accomplishments that we're proud of

  • Real regressions in pip's version parser, caught from the parent commit as if the PRs were still open, on two backends.
  • A 363× complexity regression inside a merged "109% faster" AI-written PR.
  • Every published number reproduces from a recorded image UUID and seed, and every finding ships with a verified regression test.
  • Baselines that show what the model adds, and limitations that are published rather than buried.

What we learned

Lightning executes and Ultra reasons: in our ledger Lightning made about 79% of the Token Factory calls for about 1% of the spend, and Ultra's planning and adjudication were the rest. Reasoning controls matter: medium_effort via chat_template_kwargs halved Ultra's reasoning tokens on the planner and kept its quality. An adjudicator must quote its evidence, and even then its label varies between runs on borderline classes, so a split vote becomes needs-human rather than a verdict. And Breakwater falsifies; it doesn't prove. A PASS means these probes, on this budget, found nothing.

What's next for Breakwater

  • Audit open PRs before they merge, through the Action.
  • Cite the public spec (PEP 440, project docs) in adjudication to settle the borderline "intended" labels.
  • Weight slowdowns by how often each input shape occurs in the wild.
  • Confirm finalist slowdowns with wall-clock timings alongside the instruction counts.

Built With

Share this project:

Updates

Submission history