MutAgent: AI-written tests graded by mutation score

• 

  How MutAgent works

Inspiration

LLM-written tests often pass without testing anything. "Tests pass" and "high coverage" are easy to fake. Mutation testing asks the question that matters: if I plant a small bug in the code, does any test fail? I made that question the agent's grade.

Line coverage hides the same problem. The tests that ship with the libraries in my demo run 100% (cachetools), 98.8% (inflection), 75.6% (slugify) and 64.7% (toy module) of the module's lines, yet catch only 89.7%, 90.8%, 55.4% and 34.1% of the planted bugs. Even at 100% coverage, 6 of 58 bugs got through.

What it does

You point mutagent at a Python module and its existing tests.

1. It plants small bugs (mutants) of 12 kinds: flipped comparisons (including in and is), swapped arithmetic, and/or swaps, negated conditions, return x replaced by return None, changed constants, and inside functions removed calls, removed raise statements, swapped break/continue, dropped minus signs and assignments replaced by None.
2. It runs the test suite against every mutant. Mutants that no test catches are "survivors": real gaps in the tests.
3. NVIDIA Nemotron Super reads the surviving mutants as diffs and writes tests aimed at them.
4. New tests are kept only if they (a) pass on the original code and (b) strictly raise the mutation score. Tests that fail on the original code are dropped one by one, not the whole file; the model then sees the real pytest output once and fixes its wrong expectations, and that output is carried into the next round so the same mistake isn't repeated.
5. It measures line coverage (coverage.py) before and after and shows it next to the mutation score, so the gap between "lines run" and "bugs caught" is visible.
6. When it stops, Nemotron Nano writes one sentence per surviving bug: what check is missing, or why the bug probably can't be detected at all.

Ways to use it:

• CLI: mutagent run on one module, mutagent scan --project . --discover to score a whole project, --fix to let the agent work on every weak module.
• CI gate: mutagent score --min-score 85 fails a pull request when the tests are too weak (no API key needed). As a GitHub Action it can plant bugs only on the lines a pull request changed (diff-base), post the score table as a PR comment, and --badge writes a README badge.
• Clean output: --minimize drops the generated tests that catch nothing the others miss, and checks the score stays the same.
• Reports: mutagent report turns a run into a readable Markdown report.
• Nebius Sandboxes: --sandbox runs every pytest (the project's tests, the model's new tests, and one run per planted bug) in isolated Token Factory Sandboxes instead of on your machine, with the planted bugs tested in parallel. A failing sandbox stops the run; it is never counted as a caught bug.
• Web: the hosted demo replays real recorded runs and offers capped live runs on the same fixed examples. Your own code never goes to my server: it runs locally or in your CI, with your own key. mutagent serve --local adds a "Your own code" panel to run the agent on any project on your machine. Every planted bug is a circle: hollow = slipped through, filled = caught. You watch the circles fill in round by round, see the bugs that got away with their explanation, and copy or download the new tests.

Results

Real runs against Nebius Token Factory with NVIDIA Nemotron models, every test run inside Nebius Token Factory Sandboxes, best of 3 attempts, with Tavily documentation lookups for the real libraries. Each row compares the agent with a baseline that I built in: the same loop, model, attempts, docs, repair steps and acceptance rule, but the model is never told which bugs survive.

On modules the shipped tests barely reach, mutagent scan --fix took slugify special.py from 77.8% to 100%, cachetools keys.py from 36.8% to 100%, and cachetools _cached.py from 23.8% to 96.2% (single attempts, earlier set of planted bugs).

What I take from it: the loop reliably turns weak tests into strong ones, and every kept test is proven against the real code. In this recording, showing the model which bugs survive won all four comparisons, by 2.4 to 8.6 points; on cachetools the baseline could not improve the score at all. Earlier single-attempt recordings were mixed (ahead 3, tied 3, behind 2 out of 8), so I report it as a real but modest edge, not a guarantee. It also costs about 23% more tokens. Trimming removed 30-82% of the kept tests without losing a caught bug. Failures and rollbacks are kept in the reports, not hidden.

How I built it

• Python, a small AST-based mutation engine (isolated in one file so it can be swapped for mutmut or Stryker). Every run happens in a temporary copy of the project; the original source is restored after every mutant.
• NVIDIA Nemotron models through Nebius Token Factory (OpenAI-compatible API): nvidia/nemotron-3-super-120b-a12b writes and repairs tests, nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B explains the surviving bugs.
• A hard per-run token budget, so a public demo can't drain credits.
• FastAPI and a single-page UI, deployed with Docker on Render. The hosted demo only runs the fixed examples (recorded runs, plus live runs capped by rounds, tokens and runs per day), because running strangers' code and a model's tests on a public server with my key would be a security and cost risk. The "Your own code" panel only runs on 127.0.0.1, never writes into your project, and can send every test run to Nebius Sandboxes.
• Model-written tests are screened (a denylist) and, by default, run in subprocesses with timeouts and resource limits. That is not a sandbox, and the README says so. With --sandbox they run in Nebius Token Factory Sandboxes instead, which is real isolation.
• CI on GitHub Actions (Python 3.10 and 3.13), 149 tests that use a fake model.

Challenges

• Nemotron Super often answered with invalid Python (16 times in the final recording) or long think-aloud comments inside the test file. Invalid replies get one repair attempt; think-aloud comment blocks are stripped at the end, with a syntax-tree check that the tests themselves are unchanged.
• Real projects use a src/ layout; the first version tested the installed library instead of the mutated copy. Fixed by putting the mutated copy first on the path and adding --tests to run only relevant test files.
• The reasoning model sometimes spent its whole reply budget thinking and returned no code. I raised the reply limit, tightened the prompt and added a switch for thinking.
• One wrong expectation used to throw away a whole file of good tests. Now only the failing tests are dropped, and the model repairs them from the real pytest output. On cachetools _cached.py, carrying that output into the next round took the agent from 56% to 96%.
• Models write tests in many shapes: parametrized tests, test classes, unittest classes, replies with an unclosed code block or no code block at all. Each of these broke the loop once; each is handled now, and the log says why whenever a test can't be dropped.
• The model kept trying to "call the mutant" inside a test. The prompt now says plainly that tests only ever run against the real code.
• A cheap triage step with the small model sounded good but twice hid a bug the agent could catch, so it is off by default.
• Windows saved generated tests in a legacy encoding and crashed on a non-ASCII character; everything is now UTF-8.

Accomplishments I'm proud of

• Works on real, unmodified open-source libraries (cachetools, python-slugify, inflection), not just a toy.
• A built-in baseline, so the result says what the survivor feedback adds, and I report it even where the baseline wins.
• Real isolation: with --sandbox, model-written code never runs on my machine, and planted bugs are tested in parallel in Nebius Token Factory Sandboxes.
• It is a tool, not just a demo: CLI, whole-project scan, CI gate, Markdown reports, and a local web panel for your own code.

What I learned

Reasoning models can burn their whole reply budget thinking, so reply limits and prompt discipline matter. A strict acceptance rule only works if it isn't all-or-nothing. And the most useful thing to tell a model is often the real output of the code it got wrong.

What's next

Swap in a full mutation engine, support more languages, use Sandboxes by default on the hosted demo, and let the agent open a pull request with its tests.

Try it yourself

1. Open the demo URL. If the page is slow, wait about a minute: the free host was asleep.
2. Pick a library, keep "Recorded run" and press Play to watch a real run, next to the baseline score.
3. To run it yourself: git clone https://github.com/Sarmax04/mutagent, cd mutagent, pip install -e ".[dev,web]", then mutagent score --project examples/demo_project --target pricing.py (no API key needed). With a Nebius key in .env: mutagent run --project examples/demo_project --target pricing.py --explain, or mutagent serve --local and choose "Your own code".

Built With

• coverage.py
• docker
• fastapi
• github-actions
• nebius-sandboxes
• nebius-token-factory
• nvidia-nemotron
• pytest
• python
• render
• tavily