For AI reviewers
- Problem: coding agents ship a fix and its tests in one PR; green CI can come from tests that cannot fail.
- User: maintainers and teams merging agent-written Python PRs.
- Mechanism: Nemotron Ultra on Nebius Token Factory writes one proof test from the linked issue; a plain harness, not a model, decides.
- Measured (42 live Token Factory runs): 10 of 21 real fixes proven, 0 of 9 wrong patches accepted, 0 of 12 gates opened on rejected Nemotron fixes, 28 of 31 mutants killed, $2.22 total. Evidence: https://github.com/aarav1656/latch/blob/master/latch/runs/eval-summary-repair.json
- Real PRs with Latch receipts: https://github.com/aarav1656/latch-demo/pulls
- Differs from CodeRabbit, qodo-cover, Slopgate, AegisPR: executes red on base, green on fix, red again on revert, plus mutation of the changed lines, and measures wrong-patch acceptance.
How it works
Agent PR ──► Latch reads the linked issue
│
▼
Nemotron Ultra (Nebius Token Factory) writes ONE proof test
│
▼
┌────────────── harness, no model ──────────────┐
│ base commit test FAILS with an assertion │
│ fix commit test PASSES, repo suite green │
│ 5 reruns same answer every time │
│ fix reverted test FAILS again │
│ mutants >= 60% of changed-line mutants │
│ killed │
└───────────────────────────────────────────────┘
all hold │ │ any fails
▼ ▼
GATE OPEN GATE SHUT
JSON receipt + named verdict, e.g.
PR comment FIX_FAILS_TEST
When the gate stays shut
GATE SHUT ──► failing proof test goes back to Nemotron Ultra
▲ (tests are read-only to the model)
│ │ new patch
└──── harness re-runs every check ◄┘ up to 3 rounds
Inspiration
A developer on X described an independent verifier reversing one condition so the code counted finished tasks instead, and all 12 tests still passed. Another verifier reported five deliberately broken changes that left all 1,621 relevant tests green.
Coding agents now put the fix and the tests for it in the same pull request. CI goes green, the reviewer sees green, and it merges, though nobody has checked whether those tests could ever fail. Most advice stops at "make the agent run the tests", and running a test that can't fail proves nothing. What we wanted was a gate that asks the opposite question: would this test have caught the bug, and would it catch the fix being undone?
What it does
You maintain a Python library. An agent opens a PR that says "closes #3298", with a code change and a new test, and CI is green. You can read the diff, but you can't tell whether that test would ever have caught the bug.
Run latch prove <PR URL>, or add the Latch GitHub Action and it posts the verdict on the PR. Latch treats the linked issue as the description of correct behaviour. Nemotron Ultra on Nebius Token Factory reads the issue as the spec and the diff to know where to aim, and writes one focused pytest. From there a harness decides, and the model gets no vote:
- The test has to fail on the base commit, either on an assertion or on the issue's own exception caught and reported with
pytest.fail. A test that errors at import, setup or collection doesn't count. - It has to pass on the fix, with the repository's own suite still green.
- It has to give the same answer over 5 reruns on each side.
- The harness reverts the fix, and the test has to fail again.
- The harness mutates only the lines the fix changed, and the test has to kill at least 60% of those mutants. A mutant is a copy of the fix with one small change, such as
==flipped to!=; a test that really pins the fix fails on it.
Pass all five and the gate opens with a VALIDATED verdict. Anything else gets a named refusal: FIX_FAILS_TEST (the test is sound and the fix doesn't satisfy it), INVALID_TEST (Latch couldn't build a sound test, which says nothing against the fix), NO_REGRESSION_FOUND, FLAKY or MUTATION_SURVIVED. Every run writes a JSON receipt with both commit hashes, the generated test, both test outputs, every mutant and its diff, and every model call with its completion id, tokens and cost.
Here's a real one. Click issue #3298: help text crashes with a ValueError when an option's default is a semver.Version and show_default=True. Ultra wrote test_semver_version_default_show_default, which builds that exact option and asserts the help text shows the default. On base commit 04ef3a6 it failed 5 times out of 5, and Lightning confirmed the failure is the issue's own ValueError. On fix d340b0c it passed 5 times out of 5. It failed again when the harness reverted the fix, and it killed 2 of 3 mutants. The Ultra call that wrote the test cost $0.0060, and the whole receipt $0.0066.
We ran the gate 42 times on Token Factory: 21 real bug fixes (9 upstream fixes from cachetools, boltons, tabulate, toolz, packaging, more-itertools and click, plus 12 SWE-bench Lite instances, all from sympy), 9 hand-built plausible-but-wrong patches, one per upstream case, and 12 runs where Nemotron writes the fix itself and the instance's hidden tests judge it.
| Measured on Nebius Token Factory, 42 runs | Result |
|---|---|
| Wrong patches failed by a valid proof test | 6 of 9 |
| Wrong patches the gate let through | 0 of 9 |
| Real fixes proven through all five checks | 10 of 21 |
Correct fixes Latch blamed (FIX_FAILS_TEST) |
1 of 21 |
| Mutants of proven fixes killed by the proof test | 25 of 25 (one proven fix had no line to mutate) |
Mutants killed across every run with mutation data, including the MUTATION_SURVIVED fix |
28 of 31 |
| Nemotron-written fixes on the 12 sympy instances the gate opened on | 0 of 12 (hidden tests pass 0 of 12) |
| Flaky proof tests | 0 of 12 checked |
| All 42 runs, at list prices | $2.22 ($0.22 per proven fix) |
Three of the 9 wrong patches were held shut because Latch couldn't build a valid test for that case, which is why the 6 of 9 row comes first. A second reviewer judged two of our hand-built wrong patches (cachetools-423 and click-3360) equivalent to the real fix, even though the upstream tests reject both; leave those out and it's 5 of 7 failed by a valid test, 7 of 7 shut.
The 11 real fixes that didn't get VALIDATED break down the way a merge gate should: 8 INVALID_TEST, where Latch says it couldn't build a test rather than blaming your fix, 1 NO_REGRESSION_FOUND, 1 MUTATION_SURVIVED, and a single FIX_FAILS_TEST (sympy-24909), the one time Latch pointed at a correct fix. When Nemotron wrote fixes for the sympy instances itself, the gate stayed shut on all 12, and the hidden SWE-bench tests agree that none of those fixes was correct.
AegisPR, qodo-cover and CodeRabbit publish no wrong-patch rejection rate; Slopgate reports 0 of 14 proven rewrites breaking hidden tests, on a different task (removing slop while keeping behaviour). AegisPR runs a proposed test on base and PR commits, qodo-cover writes tests that pass and raise coverage, and CodeRabbit comments on the diff. Latch anchors the test to the maintainer's issue, reverts the fix to check that the test notices, and counts how often a wrong patch gets through.
Latch ships as a CLI (latch prove, latch loop, latch eval), a GitHub Action that keeps one comment current on each PR, and a site with shareable receipts at /r/owner/repo/pr. latch loop hands Nemotron a failing proof test and lets it rewrite the code while the harness blocks any edit to the tests; on our constructed bookings regression (latch/runs/loop-demo.json) its first rewrite opened the gate. Three python-tabulate PRs in aarav1656/latch-demo carry real Latch comments: one validated, one wrong patch held shut, one where mutants survived.
How we built it
Python 3.10+, standard library only on the gate path, so it runs anywhere pytest runs.
Two Nemotron models split the work. nvidia/Nemotron-3_5-Lightning reads fast: it maps the repository and triages whether a base failure actually reproduces the issue. nvidia/Nemotron-3-Ultra-550b-a55b writes the proof test, any test repair, and the fixes in the loop. In the 42-run benchmark Ultra answered 109 calls at a median 12.6 s and Lightning 60 calls at 6.1 s.
Ultra holds the test writer's seat because it won a head-to-head. We put nvidia/Nemotron-3.5-Super-VL there for 5 cases, each with its real fix and a wrong patch. Super proved all 5 real fixes for $0.0059 per proof against Ultra's $0.0087, and it let packaging-810's wrong patch through: its test only checked dev-release versions, while the upstream test file fails 5 tests on that patch. Ultra's tests failed all 5 wrong patches. For a gate whose job is refusing wrong fixes, that settles it, and latch eval run --test-model swaps the writer for anyone who prefers the cheaper trade.
The harness pulls each commit out with git archive into a temporary directory, so it never checks out or touches your working tree. Mutation testing walks the AST of the changed lines only: flipped comparison boundaries, negated conditions, and/or swaps, off-by-one integer constants and a None return, on exactly the lines the fix touched.
latch eval pins models, prompt hashes, dataset revision and repository SHAs in a manifest. latch eval oracle runs the human-written upstream tests on every case first, at $0 and with no model, to confirm they fail on base, pass on the real fix and reject the wrong patch. Ours ran hours before the benchmark and passed 21 of 21 and 9 of 9. Then latch eval run writes the summary, a report and every receipt.
Challenges we ran into
Our first live benchmark proved 7 of 21 real fixes and failed only 3 of 9 wrong patches with a valid test. Reading the receipts one by one, the gate was doing its job and the tests were the bottleneck: Ultra often let the bug's exception escape uncaught, so the test crashed on base instead of failing, and the harness rightly refused to count it.
We added one repair round. Ultra sees only its own test's error output and rewrites the test, and the rewrite goes through all five checks from scratch. The rerun proved 10 of 21, doubled the wrong patches failed by a valid test from 3 to 6, and held packaging-810's wrong patch shut after the first run had let it through. Both runs are committed side by side with a per-case diff, so anyone can see which verdicts moved.
Accomplishments that we're proud of
- 0 of 9 wrong patches accepted, 6 of them failed by a sound test, on a benchmark whose answer key was checked with the upstream tests before any model call.
- 25 of 25 mutants killed across the fixes Latch proved.
- A real upstream receipt you can open: Click issue #3298 and fix #3299, proven on Token Factory for under a cent.
- Every number on the evidence page is read from a committed file, and anything missing renders as "not recorded" instead of a guess.
What we learned
A test can fail on the base commit, pass on the fix, and still let a mutated fix through. That's the MUTATION_SURVIVED verdict, and it fired on a real tabulate fix in our benchmark, where the proof test killed only 2 of 5 mutants.
We also learned where the model belongs. Nemotron is good at reading an issue and writing a pointed test; deciding whether that test proves anything is a job for pytest exit codes, and every verdict in Latch is one.
What's next for Latch
- Live proofs from the hosted site on a Nebius serverless endpoint, using the container deploy script in the repo (
latch/web/deploy.sh). - Non-sympy SWE-bench Lite instances from django, requests and flask in the benchmark, next to the 12 sympy ones.
- A SWT-Bench Lite slice, so our fail-to-pass rate sits beside Otter (31.4%) and AssertFlip (43.6%) on the same instances.
- The GitHub Action offered as a required check to the maintainers of the seven libraries already in our benchmark.
Log in or sign up for Devpost to join the conversation.