-
-
What counts as evidence, and what explicitly does not
-
httpx 0.28.0: twelve of sixteen claims supported
-
It checks its own release notes in its own CI
-
Unverified, not false: git cannot see a promise about a future API
Inspiration
Every tool in this space writes release notes from the diff. That is the easy direction, and it is the one nobody checks: a generated note is trusted because a machine made it, and a hand-written note is trusted because a person made it. Neither is verified against what actually shipped.
I wanted the other direction. Take the note that exists and ask what evidence supports each line.
What it does
Point it at a repository and two refs. It prints three blocks, always all three:
- supported — these commits and these files do what the line says;
- shipped without a mention — this change went out and no line covers it, which is how a breaking change reaches users quietly;
- no evidence found — this line claims something the diff does not contain.
It exits non-zero on the third, so it sits in CI as one line and fails the release before the notes are published.
On a real release — httpx 0.27.2 to 0.28.0, checked against that project's own changelog — it supports 12 of 16 claims, finds 4 silent changes and 4 lines with nothing behind them. That is not an accusation against httpx. It is what a careful, human-written changelog looks like when you ask a machine for receipts.
How I built it
Python 3.11 and the standard library. No dependencies, no network, no API key: the evidence is git on the machine running it.
The matching rule is the whole design. A claim is supported only when something concrete ties it to a commit: an issue or PR number, a command-line flag, a filename, an identifier the diff defines, or a code-like token in the commit subject such as verify=False. Shared ordinary English is not evidence — "improves the export" and a commit called "tidy up exports" have a word in common and prove nothing.
That rule misses real matches. Missing them is the acceptable failure here, because the cost of a false supported is a release note nobody checks again.
Challenges I ran into
The first version was too strict in one direction and too loose in another. It refused a true line because the note said hours.py and the path token was hours — a filename spelled out in a note is the strongest signal a human ever leaves, and it was being thrown away. And it ignored commit subjects entirely on the grounds that they are prose; but Fix \verify=False`, `cert=...` case` is not prose, and refusing to read it lost an obvious match on a real httpx release.
Both fixes are narrow and testable, which is the point of having tests that build git repositories on the spot.
Accomplishments that I'm proud of
It is honest about the line it cannot cross. "We are working towards a simplified SSL configuration API" is reported as unverified, not as false — git cannot see a promise. The report prints what it searched for next to every unsupported line, so a person can judge the judgement.
And it checks its own release notes in its own CI.
What I learned
That the useful question is not "can a model summarise this diff" but "what would count as proof". Once proof was defined — an issue number, a flag, a filename, an identifier — the tool stopped needing a model for the verdict at all. The model is behind --explain and only turns an established match into a readable sentence; with no key the report still prints.
That boundary came from a measurement in an earlier project, where a model asked to do arithmetic corrected a right answer into a wrong one inside a single reply. Deciding what a model is not allowed to decide turned out to be the design work.
What's next for Changelog Check
Matching against tests rather than commits, so "fixed the crash on startup" can rest on a test that used to fail. Reading notes straight from the GitHub release API. And a mode that drafts the missing lines for the silent changes, clearly marked as a draft.
Honest limits
It reasons about lines and commits, not behaviour. A squashed history gives it less to work with, and it says so. Weak path matches can attach a plausible commit to an unrelated line; they are ranked last and labelled, so a reader can see the difference.
Built on 24 September 2026, inside the submission period, with the Devpost Learn skill pack: devpost/scope.md, prd.md and spec.md were written before the code.
Log in or sign up for Devpost to join the conversation.