Receipts

Inspiration

A repository is full of claims aimed at a stranger. A licence badge. A coverage percentage. A supported Python version. An example in the README. A line in the changelog.

Every one of them is written once and then drifts. Tests check behaviour. Linters check style. Nothing checks the claims.

I had four small checkers, each answering one of these questions on its own. Separately they are scripts a maintainer has to know about, install and remember. This is the question they share, asked once.

What it does

./run.sh --repo . --html receipts.html

Check	The claim	What backs it
badges	"MIT licensed", "coverage 100%", "Python 3.7+"	the LICENSE file, the packaging metadata — or nothing
documentation	"call it like this"	the modules, functions, methods and flags the code defines
python floor	requires-python and the trove classifiers	the syntax the shipped code actually uses
release notes	"this release fixed X"	the commits in the range

Three severities — contradicted, nothing behind it, no longer matches — and a fourth list that matters as much: not checked, with the reason. Only contradicted changes the exit code, so an audit can sit in CI without failing a build over a stale badge.

Nothing is imported, executed, installed or fetched. A badge's claim is read out of its own URL, because auditing a claim by loading the thing that makes it would defeat the point.

The badge check, which is the new one

Badges are the loudest claim in a project and the least examined.

badge	checked against	verdict
badge/License-MIT-blue	the LICENSE file	checkable
badge/python-3.7%2B	requires-python	checkable
badge/coverage-100%25	nothing — the number is in the URL	nothing behind it
codecov.io/.../badge.svg	a server	listed, not judged

Found in public repositories

Forty-five projects audited, every finding verified by hand against the file it accuses before it was written down. Three did not hold up:

• urllib3 — a badge reading coverage 100%, with the number written into the badge URL. Nothing measured it and nothing can make it false. Not filed upstream: urllib3 does enforce full coverage in CI, so the claim is currently true and an issue would be noise.
• python-docx — classifiers advertising Python 3.7 and 3.8 while requires-python is >=3.9. pip enforces requires-python and ignores classifiers, so PyPI shows support that pip refuses to install. Filed: python-openxml/python-docx#1620
• pdfplumber — the opposite direction: python_requires=">=3.8" with classifiers starting at 3.10, so pip installs on two versions the project no longer claims. Filed: jsvine/pdfplumber#1400

Everything else came back clean, which is the result that matters: these are well-kept projects, and a tool with an opinion about them would simply be wrong.

How I built it

Planned before written — devpost/scope.md, prd.md, spec.md and a checklist are in the repository, dated before the code. The four checkers are vendored as libraries under src/, and promises.py is the only file that knows their internals: every check returns the same Section, so a checker can change without the report changing with it.

Challenges I ran into

A correction, not a feature. I had read the Python floor as the lowest number stated anywhere, and written in the report that "pip honours the lowest". Both were wrong: pip enforces requires-python and nothing else; the classifiers only advertise. python-docx is the live case, and the fix is in this repository and in the standalone checker with three tests.

A changelog should not mention everything that ships. Run against httpx, the notes check reported a dependabot bump and a test-only commit as unmentioned changes. A commit that edited the changelog cannot have shipped without a mention in it. All three kinds are now excluded.

A test that could not fail. I wrote a test asserting the tool never prints another project's warnings to stderr. It passed with the fix reverted, because pytest captures warnings before they reach stderr. It now runs the CLI as a subprocess and reads real stderr; reverting the fix turns it red.

Accomplishments

• 36 tests, green on the Python this promises (3.9) and a current one.
• 17 of them adversarial: an empty directory, no Python at all, invalid TOML, bytes that are not UTF-8, a symlink loop, a dangling symlink, a 200,000-character line.
• demo/crash-hunt.sh runs every repository in a directory through all three output forms and exits non-zero on any stderr or any exit code above 1. 69 clones come back clean.
• Two findings accepted as worth reporting and filed upstream.

What I learned

Every bug in this thing was found by running it against real repositories and then checking the accusation by hand — never by reading the code and thinking about it. Six of them today, including three in the checkers I had already written.

Honest limits

• It checks the claims it knows how to check. A repository asserts plenty more, in prose, on a website, in a video, and this reads none of it. That is why the report ends with what it did not ask rather than with a score.
• A badge that resolves remotely is listed and not judged: its claim lives on a server.
• The release-notes check needs at least two tags, so a shallow clone cannot answer it.
• Python only.

Built with

Python 3, ast, pytest, GitHub Actions. No runtime dependencies, no network.

Built With

• ast
• github-actions
• pytest
• python