falsegreen

Mutation testing for the guards in your pipeline: it breaks a good artifact on purpose and tells you every time your verify step still says "pass".

Comment

Inspiration

I run an automated pipeline that writes reports, slides, documents, audio and video, and every stage ends in a verify step. Over two months I kept notes on every time it shipped something wrong. Almost none of those failures were red. They were green: the guard ran, printed passed, the file opened normally, and something that should have been in it was missing. A length check that was never given a rule. A required field filled with the template's instruction text. A summary that said 3 items while the body held 1. A big number written as 2.2E+09 that a type check accepted.

Code has mutation testing. Verify steps usually don't. Nobody feeds their own guard broken-but-plausible files, so nobody finds out what it is blind to until it ships.

What it does

falsegreen takes a known-good artifact and your guard (any shell command that answers with its exit code), then:

  1. Baseline: checks that your guard accepts the clean artifact. If it doesn't, the run stops as INCONCLUSIVE instead of reporting nonsense.
  2. Mutate: applies up to twelve failure families taken from real incidents: empty-input, placeholder-as-content, occlusion, overflow, declared-vs-rendered, metadata-identity, stale-cache, numeric-truncation, dead-regex, silent-track, self-referential-check, structured-overrides-body. Formats: text, Markdown, JSON, CSV, SVG and WAV.
  3. Judge: if the guard rejects a mutant, that family is CAUGHT. If it still exits 0, that is a FALSE GREEN.
  4. Report: a terminal table, JSON, or Markdown, ending in one line: "Your guard is blind to N of M failure families", with a link to how to fix each one in FAILURE_MODES.md.

It refuses to produce its own false greens. If a mutator did not actually change the file, or the change doesn't pass a semantic check for that family, the result is INCONCLUSIVE, never green. A mutation tester that silently does nothing would be exactly the failure it is meant to find.

On the included example, a careful hand-written guard that checks required fields, item count, item text, number types and attachments is blind to 7 of 8 applicable families. The hardened reference guard catches 8 of 8. falsegreen bench runs both reference guards over 12 fixtures: naive 0/12, hardened 12/12, with 24/24 expected verdicts met.

How we built it

  • Built with Claude Code and the Devpost Learn Skill Pack (npx skills add challengepost/learn-ai-basics --all -y). The skills drove the process: scope.md → prd.md → spec.md → a sliced build checklist, one commit per slice. All three planning documents are in the repository under devpost/.
  • Plain Node.js (20+), ES modules, zero dependencies. Your guard can be written in any language because falsegreen only reads its exit code.
  • 43 tests (node --test), including tests that prove the three-state self-check: a mutator that does nothing, one that lies about what it changed, one that throws, and a guard that hangs all come back INCONCLUSIVE.
  • A test that feeds every mutator's verifier the original file found a real bug during the build: the dead-regex verifier was trusting its own list of patterns instead of the mutated file. That is the same failure the tool exists to catch, and we fixed it.

Challenges

  • Proving a mutant is really broken. Changing bytes is easy; showing the change is the failure we claim is not. Each family has its own check (for occlusion, that the covering shape really overlaps the text box).
  • Being honest about coverage. A family that can't apply to a file type is reported as n/a, not counted as caught. A rough heuristic is labelled as a heuristic in the README.
  • Keeping the incidents useful without exposing the private pipeline. FAILURE_MODES.md restates each incident in general terms (symptom, why it looked green, how to reproduce, how to fix) with its date.

Accomplishments we're proud of

  • Every family in the table comes from something that really happened, with a date, not from a brainstorm.
  • The demo guard isn't a strawman: it does real checks and is still blind to 7 of 8 families.

What we learned

A passing check only means something if you have seen it fail. The most dangerous result is a pass that never had anything to check.

What's next

  • More formats: PDF, Office documents, video containers.
  • A GitHub Action that runs falsegreen against a project's own verify scripts on every pull request.
  • Community-contributed failure families, each with a dated incident.

Pre-existing work (disclosure)

All code was written during the submission period (repository history starts 2026-10-18). The failure families were generalised from the author's private incident notes (July–September 2026, prose only, no code). See PREEXISTING.md.

Links

Built With

Share this project:

Updates

Submission history