KillScore

Inspiration

In 2026 roughly a quarter of merged commits carry AI traces. Coding agents write the tests too, and the tests reach 100% line coverage on the first try. That number is what release engineers use to approve a merge. We flipped one comparison operator in a payment function and all 40 AI-written tests still passed. Coverage never moved. The metric we gate on cannot see the failure mode that AI-written tests introduce, because a test derived from the implementation cannot disagree with the implementation.

## What it does

KillScore is a pull-request gate. It reads the PR diff, finds only the Python functions the diff touched, and plants small deliberate bugs (mutants) in them: > becomes >=, and becomes or, a constant shifts by one, an except: raise becomes except: pass. It then runs your own test suite against every mutant in an isolated subprocess. A mutant your suite fails on is killed. A mutant your suite passes on survived, and that is a bug your tests would not catch. The mutation score is the share killed. Below your threshold, the merge check fails.

For each survivor, Claude drafts one test that should kill it. Before you see that test, our runner executes it twice: it must pass on the original and fail on the mutant. Only then is it labelled VERIFIED KILL. Otherwise it is REJECTED and queued for a human. The model never computes the score, never marks a mutant killed, and never edits the code under test.

## How we built it

• FastAPI + SQLAlchemy + Postgres. Structured JSON logs, an llm_call table with tokens, cost and latency per request.
• Mutation engine on Python ast, 9 operators. Diff scoper maps unified-diff line ranges to FunctionDef nodes.
• Runner copies a worktree per mutant, runs pytest -x -q with an 8-second timeout across 8 workers, and reads the exit code. Timeouts stay in the denominator and never count as kills.
• Server-Sent Events push each verdict to a Next.js run view where tiles flip green, red or grey as they land.
• Gate policy produces a GitHub Checks payload and a TMMi-style evidence PDF with a SHA-256 over every verdict.
• Claude Sonnet via OpenRouter with a strict JSON schema for propose_test. A proposal citing a mutant id not in the run is rejected before execution.
• Three seeded demo repos: an AI-tested billing module, an auth module with 59% coverage but better tests, and a vendored real OSS package (inflection 0.5.1, MIT).

## Challenges we ran into

Getting the score to mean something. Timeouts are the obvious way to inflate mutation scores, so they are excluded from kills by design. Making the diff scoper the product: full-repo mutation takes hours; scoping to touched functions makes it seconds and makes it a PR gate rather than a CLI. Keeping the model honest: on the demo repo the verification loop accepted 9 of 10 Claude proposals and rejected one that failed on the original code, meaning the model had encoded a wrong expectation as a test. That rejection is the product working.

## Accomplishments that we're proud of

The product refuses its own AI output on camera. Coverage 100%, mutation 63%, 10 survivors surfaced with the exact line and operator, on a live system. On 20 repos with 20 planted bugs, the AI-written suite caught 6, a model reading the code caught 12, KillScore surfaced 19, and adding only runner-verified tests moved the mutation score from 75% to 97%.

## What we learned

Success-case appeal is not the same as expected value. We picked the idea whose demo cannot fail over the idea whose demo would be flashiest.

## What's next for KillScore

GitHub App with the real Checks API, JavaScript/TypeScript operators, Slack alerts, and self-hosted runners for private code.

Built With

• apache-2.0
• ast
• docker
• fastapi
• mutation-testing
• nextjs
• openrouter
• postgresql
• pytest
• python
• react
• sqlalchemy
• sse
• tailwind
• typescript
• weasyprint