Feathers

Inspiration

Every developer eventually inherits code with no tests. Michael Feathers' Working Effectively with Legacy Code gives the answer — characterization tests that pin current behavior before you change anything — but writing them by hand is slow, and you often don't understand the code well enough to know what to assert yet. AI test generators try to help, but they hallucinate expected values, so you can't trust the tests they write. We wanted a tool that uses AI where it's strong and never where it's dangerous.

What it does

feathers analyze ./repo runs a five-stage pipeline: it walks the repo's AST to inventory every function and method; GPT-5.6 categorizes each unit and scores its risk; for each unit GPT-5.6 proposes diverse inputs and Feathers executes the real code in a sandboxed subprocess to capture the true output (return value or raised exception) and emit pytest tests asserting exactly that; GPT-5.6 then drafts higher-tier tests (unit, acceptance, e2e) and a phased testing plan; finally every generated test is run and a self-contained HTML report presents the categorization, risk map, characterized behavior, phased plan, and per-test status.

The key idea is "LLM proposes, execution disposes." GPT-5.6 never decides what a test should expect — it only imagines inputs. Feathers runs the actual code to get the ground-truth output. So characterization tests are green by construction and trustworthy: the expected values come from your code, not a guess. That single choice is what separates Feathers from generic AI test generators.

How we built it

The entire pipeline was built by Codex, agentically, from a detailed test-driven implementation plan. Codex built nine focused modules — AST ingester, sandboxed subprocess executor, characterization engine, GPT-5.6 client, categorizer, generator, verifier, HTML report renderer, and CLI — each one test-first: write the failing test, run it red, implement, run it green, commit. It was Codex doing genuine multi-step engineering across a whole codebase, not snippet autocomplete, and the tool that tests other code ended up thoroughly tested itself.

GPT-5.6 also runs inside the product at four points: categorizing units and scoring risk, proposing characterization inputs, authoring the phased testing plan, and drafting the higher-tier tests. The reasoning is GPT-5.6; the ground truth is the executing code. Stack: Python 3.11+, pytest, the OpenAI API (GPT-5.6), Jinja2 for the HTML report, and a click CLI.

Challenges we ran into

Running untrusted code safely — characterization means executing code you don't trust, so every execution runs in an isolated subprocess with a hard timeout, and a hanging or crashing unit degrades into a captured result instead of taking down the tool. Keeping AI honest — we deliberately restricted GPT-5.6 to input proposal and drafting, never expected values, so the trustworthy tests stay deterministic. And graceful degradation — a unit that raises is characterized as raising, a bad proposed input is skipped, and a failing drafted test is flagged rather than fatal. Nothing aborts the run.

Accomplishments that we're proud of

Characterization tests that are green by construction and provably run. Self-verifying multi-tier generation with honest pass / needs-review labeling. And a complete product experience — CLI plus a polished HTML report — not a proof of concept.

What we learned

Let the model do what it's good at — imagining inputs, drafting, classifying — and let execution decide the truth. That division is what keeps AI-generated tests trustworthy instead of plausible-but-wrong.

What's next for Feathers

Language adapters beyond Python. Mutation-testing feedback to strengthen proposed inputs. And CI integration: run Feathers on a pull request and post the report as a check.

Built With

• codex
• gpt-5.6