Red First

• 

  Red First run against its own test suite, in its own CI

• 

  The four verdicts, and why a shared English word is not evidence

Inspiration

A test suite that passes is treated as proof the code works. It is not proof — it is a claim. Coverage does not settle it either: coverage tells you a line ran, never that anything checked it. I have shipped seven apps on the strength of suites I never audited, and I wanted to know which of those green checks were real.

What it does

Red First empties one function at a time and re-runs only the tests that reach it. Four verdicts, printed every time:

• unguarded — every test that reaches it still passed. It looks tested, it counts as covered, and nothing in the suite would notice if it stopped working.
• unreached — no test executes it at all.
• crashed — a test failed, but on an exception rather than an assertion. The suite noticed by accident; nothing in it says what the function should do.
• guarded — a test asserted something that stopped holding, and it is named.

It exits non-zero on unguarded, so it fits in CI as one line.

On real repositories: tinydb, 101 functions, 19 seconds — 39 guarded, 56 noticed only by accident, 6 unreached, 0 unguarded. python-tabulate, 73 functions — 29 and 43. The finding is the split, not a bug list: a tool that always finds something is not measuring anything.

How I built it

One pass of the suite under coverage.py with dynamic_context = test_function gives a map from function to the tests that execute it. Then, per function, an AST rewrite replaces the body with return None (a generator becomes yield from () so it stays a generator), and only the mapped tests run, in a fresh subprocess, against a copy of the repository. The reason a test failed is read from a small pytest plugin rather than guessed from a traceback — that distinction between an assertion and an exception is the whole point of the crashed verdict.

A model appears in exactly one place, behind --explain: for a function nothing defends it proposes one test, and the proposal is run twice before you see it. It must pass against the real function and fail against the emptied one. If it cannot tell them apart it is dropped, and the report counts how many were dropped. The model proposes; the mutant decides.

Challenges I ran into

Three bugs that were invisible from the outside, each now a test:

• The bytecode cache. CPython validates a .pyc against the source's (mtime, size), both at one-second resolution. Two mutants of similarly shaped functions collided on both, the tests imported the previous mutant, and verdicts changed depending on what ran before them.
• Plugin shadowing, then a PYTHONPATH leak. The reporting plugin was first imported from the package, which a project with the same package name shadowed; moving it to PYTHONPATH then leaked that variable into the subprocesses the tests themselves start, which repaired the mutant. The plugin now lives inside the copy being examined.
• Parsing pytest's output. Node ids were read off --collect-only; a project that sets -v prints a tree with no node ids in it, and every function came back unmatched. The plugin reports them directly now.

Accomplishments that I'm proud of

It found something in its own tests. The first self-run called one function unguarded; checking the claim by hand showed the tests did fail on that mutation, so the verdict was wrong — and the reason was the PYTHONPATH leak above. The same run named nine functions no test executed. They have tests now, and Red First runs against Red First in its own CI.

Also: on two well-tested public libraries it reported no unguarded function, plainly. Refusing to manufacture a finding is the feature.

What I learned

Where a model belongs in a tool that makes claims: proposing, never deciding. A suggestion earns its place by surviving an attempt to falsify it — here, by failing against code that does nothing and passing against code that works. That check is cheap, and it turns an unverified suggestion into a verified one.

I also learned to plan first and in writing. devpost/ holds the scope, PRD, spec and checklist, written with the Devpost Learn skill pack before any code, and the boundary they drew — one mutation operator, Python and pytest only — is why this finished as a working tool rather than an unfinished framework.

What's next

Other languages and runners, more operators, a cache keyed by file hash, and a pull-request comment. Deliberately not next: writing the proposed test into the repository. The tool proposes a checked assertion; a person decides where it lives.

Built With

• apertus
• coverage
• github-actions
• pytest
• python