Chesterton

Try it: chesterton.vercel.app (click-through version: chesterton.vercel.app/?demo) · Code (MIT): github.com/ManuelGamal/Chesterton

Inspiration

AI coding agents now open pull requests that pass every test, and reviewers merge them on that green check. But a passing suite only proves the code does what someone thought to test. Benchmarks show the gap: patches accepted by SWE-bench's own tests are later proven wrong by stronger test suites such as UTBoost.

The name comes from Chesterton's fence: don't remove a fence until you know why it is there. Chesterton does the reverse test. It removes or changes each line the agent wrote, one at a time, and asks whether anything notices.

What it does

Chesterton reviews an AI agent's pull request before you merge it, and tells you what its tests would let through. For each changed line it makes a small deliberate change (a mutant), runs the full test suite, and keeps the mutants that every test still passes.

The demo shows a real case: a matplotlib pull request written by Agentless 1.5 with Claude 3.5 Sonnet, which passed SWE-bench.

1. The run. Chesterton tries 19 changes to the patch in parallel sandboxes. The tests catch 7 and miss 12.
2. The triage. Nemotron sorts the 12 misses into false alarms, dead code and real gaps. Three are real gaps.
3. The finding. Hard-coding super().set_visible(b) to False passes every test, so nothing checks that set_visible(True) actually shows a 3D plot.
4. The missing test. Nemotron Ultra writes a regression test. Chesterton keeps it only after running it: it passes on the agent's fix, fails on the mutant, and holds on the correct fix.
5. Ask live. One button asks Nemotron Super, live, why this change matters, so a judge can check the recorded answer against a fresh one.

The page then replays how the review was found: every change, which tests caught it, and which ones nothing noticed.

How we built it

Chesterton is a Python pipeline that runs entirely on Nebius Token Factory: its sandboxes run the tests, and three Nemotron models do the thinking, each sized to its job.

• Sandboxes: one seed checkpoint per pull request, with the repo and its test environment installed once. Each mutant runs in its own fork of that checkpoint, 24 at a time.
• Nemotron 3.5 Lightning proposes the mutants: small, plausible changes to the lines the agent wrote.
• Nemotron 3 Super triages every mutant the tests missed: false alarm, dead code, or a real gap (untested_invariant), and whether that gap is about safety or behaviour. A headline finding must get the same verdict 3 times out of 3.
• Nemotron 3 Ultra writes a regression test for the top finding. Chesterton runs it in a sandbox and keeps it only if it passes on the patch and fails on the mutant.
• The demo is React 19, Vite and Tailwind on Vercel. It replays recorded runs, and its one live button calls Nemotron Super through a Python function, capped at 60 calls an hour and 100 a day by an Upstash counter that fails closed.

Pipeline: seed sandbox → Lightning mutants → parallel test runs → Super triage → Ultra test, verified in a sandbox. Only the mutants every test misses go on to triage, and only the top real gap gets a test.

Challenges we ran into

• False alarms. Many changes the tests miss don't matter: the code behaves the same either way. If a reviewer clicks a finding and gets noise, they stop trusting the tool. Triage by Nemotron Super, with a 3-of-3 agreement rule for headline findings, keeps the review quiet on good code: on the fix matplotlib's developers merged, the tests catch 5 of 6 changes and the one they miss is judged not a real gap.
• Tests that lock in the bug. A generated test can pass every check and still assert the agent's wrong behaviour. In a 13-patch study, half of the first verified tests (5 of 10) did this. One prompt rule, test through the public API, cut it to 1 in 9. The rest are caught by running each test on the correct fix, which SWE-bench provides and a new pull request does not.

Accomplishments that we're proud of

On 292 AI-written patches that had already passed SWE-bench, Chesterton wrote an execution-verified missing test for 31% of them, and 80% of those tests also hold on the correct fix. The study covered 16 tasks in 6 Python libraries, was pre-registered on 2026-09-26, and ran on 2026-09-27.

• Honest science. All three studies were pre-registered and are reported as registered.
• A live, working demo that runs on the real models, with spend capped so it stays up through judging.

What we learned

• Execution beats judgement. A model's opinion that a test is good is weak evidence; running it on the patch, the mutant and the correct fix is strong evidence. Every test Chesterton offers has been run, not just written.
• Match the model to the job. A fast model proposes many cheap mutants, a mid-size model triages, and the largest model is saved for the one test that matters.
• Model answers vary, so ask more than once. One review per patch gave noticeably different results across runs, which is why headline findings need 3 of 3 agreeing verdicts.

What's next

• A check on every pull request. Run Chesterton as a GitHub check that posts the missing test as a suggested change, so the reviewer can accept it in one click.
• A substitute for the correct fix. On a new pull request there is no reference fix to run the test against, so the last "locks in the bug" cases need another signal, such as the issue text or the reviewer's confirmation.
• Beyond Python. The pipeline only needs a way to change a line and run the tests, so other languages with fast test suites are next.

Built With

• nebius-sandboxes
• nebius-token-factory
• nvidia-nemotron
• pytest
• python
• react
• swe-bench
• tailwindcss
• typescript
• upstash-redis
• vercel
• vite