-
-
thumbnail
Inspiration
What it doesInspiration
AI coding tools can write tests that pass without testing anything: a test that asserts True is green, and so is one that calls the wrong function. Reviewers then have to read every generated test. We wanted an agent whose tests carry their own proof that they cover the code.
How we built it The agent runs a PR's test suite under branch coverage inside a bubblewrap sandbox, then compares the coverage with the diff to find changed lines that no test executes. That step uses no LLM. For each gap, NVIDIA Nemotron 3 on Nebius Token Factory writes a test. It starts with Nano and moves up to Super and then Ultra only when a test fails. A test is kept only if it passes, runs the gap lines and asserts something real.
What we learned Our first eval runs showed that two failures came from our own generator: generated tests were missing imports, and the prompt didn't number its lines. After we fixed both, Nano's first-attempt success went from 25–31% to 69%. That figure is in-sample, so it overstates performance on new code.
Challenges
On new code, most failed gaps ended on a field or class name the model made up, because the prompt shows only the function under test. We couldn't install the Nebius Sandboxes CLI, so we built our own local sandbox. Being honest about our own numbers took discipline.
