MutantReview

MutantReview finds what green tests miss in AI written code and drafts stronger tests verified by execution, approved by you.

Comment

Inspiration A passing test suite can create a false sense of confidence. Code coverage shows which lines ran, but it does not tell us whether the tests would notice those lines breaking. That gap becomes especially concerning when AI writes both the implementation and its tests, because both can repeat the same assumptions.

We built MutantReview to make test quality something developers can demonstrate. We wanted to bring mutation testing directly into GitLab merge requests, helping reviewers find missing checks and giving AI-generated tests a measurable standard before they are accepted. What it does MutantReview is a GitLab Duo agent that mutation-tests the lines a merge request changes. It introduces small changes, such as replacing >= with >, and checks whether the existing tests catch them. A mutation that survives reveals behavior the tests may not adequately protect.

The agent filters out low-value findings, prioritizes boundary conditions and branching logic, and presents up to three findings for review. It then drafts tests based on the project’s written requirements.

Every proposed test must pass on the original code and fail against the targeted mutation before it is offered for approval.

Developers choose which verified tests to apply through a merge request comment. MutantReview commits the approved tests, reruns mutation testing, records the results, and adds the approved testing lesson to AGENTS.md for future coding agents.

A configurable mutation gate enforces the project’s quality threshold. The Front Desk dashboard makes findings, rejected drafts, repair history, and mutation counts visible to the team. How we built it We built MutantReview with TypeScript, Node.js, StrykerJS, and Vitest. Its architecture separates the review logic, test execution, GitLab API integration, and dashboard.

Our GitLab Duo custom flow has three stages: deterministic preparation, one AI drafting step, and deterministic verification and publication. The model proposes tests, while code controls the scope, scoring, permissions, and approval checks.

GitLab CI/CD runs the mutation review and quality gate alongside automated tests and security checks. We built the Front Desk with React and Express and added deployment configuration for Google Cloud Run.

We also use AGENTS.md as shared repository memory, allowing human-approved lessons to guide later work across coding agents. Challenges we ran into The hardest challenge was making AI-generated tests trustworthy. A test can look convincing and still miss the exact behavior it claims to protect. Requiring execution against both the original code and its mutation gave us a concrete way to reject ineffective drafts.

Mutation testing also introduces noise and computational cost. We addressed this by limiting analysis to changed lines, filtering low-value mutations, and keeping the reviewer’s findings focused.

Another challenge was safely connecting conversational approval to repository changes. We added permission checks, proposal integrity checks, and merge request revision checks so an approval applies to the code and tests the developer actually reviewed. Accomplishments that we're proud of

  • Demonstrating a gap that coverage missed. Our recorded GitLab demo achieved 100% branch coverage while the mutation gate identified an untested boundary condition and failed at a 75% mutation score.
  • Verifying proposed repairs through execution. Our automated end-to-end tests use real Vitest and Stryker runs to accept an effective boundary test and reject a plausible test that misses the mutation.
  • Evaluating the approach on real projects. Our benchmark found surviving mutations in 11 of 25 analyzed merged es-toolkit pull requests. These represent potential test gaps, rather than confirmed production bugs.
  • Reducing the mutation workload. Changed-line scoping selected 1,057 mutations compared with 4,088 generated for the same whole files, avoiding 74.1% of candidate mutations.
  • Turning approved repairs into reusable lessons. Each approved repair can improve both the test suite and the instructions future agents receive. What we learned We learned that test quality depends on the behavior a test can distinguish. Exact thresholds, alternate branches, and meaningful expected values often matter more than adding more assertions.

We also learned to give AI a focused creative role and verify its output independently. Execution provides evidence that a proposed test catches a specific change, while human review determines whether the behavior being protected matches the requirement.

Finally, we learned that developer tools need to respect attention. A few clear findings with demonstrated repairs are easier to act on than a long list of unexplained warnings. What's next for MutantReview Our next priority is completing live validation of the full GitLab approval-and-repair flow and the Cloud Run deployment.

We also plan to support additional languages through tools such as PIT for Java and mutmut for Python, improve the identification of equivalent mutations, and retrieve lessons by module so repository memory stays useful as projects grow.

Longer term, we want MutantReview to help teams measure how their test suites improve over time and make each approved repair useful to the next developer or agent working in the repository.

Built With

Share this project:

Updates

Submission history