Example Check

Inspiration

A function gets renamed. The tests still pass, continuous integration is green, and the README keeps telling a new reader to call something that is gone.

Linters check code. Documentation builders check that the docs render. Almost nothing checks that the examples inside the documentation still name code that is there — and the first person to find out is the person you least want to lose: someone trying your project for the first time.

What it does

For each fenced block in the documentation it works out what the example references — modules, symbols, methods, command-line flags, files — and compares each reference against an AST walk of the project. It prints four groups:

• matches — everything the example references is still defined, and where;
• renamed — the name moved to another module, or one close neighbour exists;
• broken — this project defines nothing of that name;
• not judged — the block had nothing checkable in it, and the reason why.

It exits non-zero on broken, so it sits in CI as one line and fails a release whose README lies. No dependencies, no network, no API key, and it never imports or executes the project under test.

$ example-check --repo .
BROKEN  18
  README.cn.md:198  method Table.add_row
      Table has no such method, nor do its base classes
  ... 17 more, one per translated README
exit=1

What counts as evidence

The design lives in what the tool refuses to look at.

in the example	checked?	why
json.dumps(...)	no	not a claim about this project
requests.get(...)	no	third party
x = 1; print(x)	no	the snippet's own variable
from pkg.mod import run	yes	names this project
table.add_row(...)	yes	table came from a class defined here
pkg --strict	yes	a flag this project declares

Reporting any of the first three would be a false accusation against a correct README. That is the expensive direction of error, so the tool is built to fail the other way and to print everything it declined to judge.

How I built it

Planned before written, with the skill pack: devpost/scope.md, prd.md, spec.md and a checklist, all in the repository and all dated before the code. The build is five small modules — documentation splitting, project surface, reference extraction, verdicts, report — and 33 tests, each of which writes the tiny repository it is about, so every assertion is about history the test itself created.

Challenges I ran into

Every one of them was a false positive, found by running against real projects and now covered by a test:

1. Inherited methods. httpx.AsyncClient.build_request is defined on a base class.
2. Generic bases. itsdangerous declares class URLSafeSerializer(URLSafeSerializerMixin, Serializer[str]) — a subscript, not a name.
3. Chained star re-exports. httpx re-exports a package that re-exports a module, so star imports had to be resolved to a fixed point rather than once.
4. Implicit flags. Typer names --output after the parameter it fills; the string never appears in the source.
5. Frameworks it cannot read. Poetry builds its command line with Cleo. Rather than call 87 documented flags broken, the tool now says outright that it cannot read that interface and judges none of them.

Each fix made the tool quieter, not louder.

Accomplishments

Thirteen maintained Python projects — rich, click, httpx, typer, black, poetry, mkdocs, sqlmodel, python-dotenv, loguru, records, itsdangerous, python-tabulate — 2,459 examples, 986 references, nothing reported.

Zero is the result that matters. These projects keep their documentation current, so a tool with an opinion here would simply be wrong. Reproduce it with ./demo/scan.sh.

And in the other direction, on a copy of rich, performing the rename a maintainer performs without thinking:

$ sed -i 's/def add_column/def add_col/' rich/table.py
$ example-check --repo rich --quiet
RENAMED  18
  README.cn.md:198  method Table.add_column
      closest match is add_col

Eighteen findings from one rename, because that example is repeated in eighteen translations of the README. Renaming a method to an unrelated word instead produces eighteen broken and exit 1.

It also found a real bug in itself on the first run: from examplecheck import docs imports a submodule, and the check was looking for a name in __init__.py.

What I learned

The same lesson as the three projects before this one, arriving from a new direction: the value is in the refusal. A tool that always has an opinion is a tool nobody trusts twice.

Honest limits

• It checks names, not behaviour. A function that still exists but now does something else reads as matches, and it cannot know otherwise.
• Names built at runtime are invisible to a syntax tree, so a module defining __getattr__ has its absent names declared unjudged.
• renamed is a guess: one candidate above the similarity threshold is a suggestion, two or more is broken with the candidates listed.
• It knows Python. The shape of the design is general; only the surface reader is not.

What's next

Other languages, and signatures — "this function exists" is a weaker claim than "you can still call it this way".

Built With

• ast
• github-actions
• markdown
• pytest
• python