Inspiration
"We are launching today an aggressive program to cut our carbon emissions by more than half by 2030, both for our direct emissions and for our entire supply and value chain."
That is a real sentence from a real corporate climate announcement, copied word for word (source). Read it once and it sounds like a commitment. Read it twice and try to work out what you would have to know, a year from now, to say whether it was kept.
You would need to know what "more than half" means as a number. You would need a baseline year to measure the half against. You would need to know which standard defines the supply-chain part. None of that is in the sentence.
Most tools that look at corporate climate language try to score whether a company is sincere. That is a judgement about a company, and it is the wrong question for software to be answering. Checkability is a property of a sentence, not an opinion about a company. A sentence either names a quantity, a target year, a baseline year, a standard and an accountable party, or it does not, and anyone can go and look.
That is the only question Pledgecheck asks.
What it does
Pledgecheck takes corporate climate-pledge prose and, for every claim in it, reports which of five checkable elements are actually present — with the span of text that matched and the disclosure standard that requires the element.
The five probes are quantity, target year, baseline year, named standard, and accountable party. Each carries a citation to the standard that asks for it: the GHG Protocol, SBTi Net-Zero v1.2, the TCFD Recommendations, ISSB IFRS S2. A claim scores 0 to 5 and lands in a band from Unverifiable to Fully Checkable.
That opening Microsoft sentence scores 1 out of 5. It names the year 2030 and nothing else.
How it works
Five nodes. A model splits report prose into atomic claims, and a model writes the plain-English explanation at the end. Everything in between is code.
The Checkability Gate is a pure function over a versioned rule pack. It has no network, no model, no prompt, and no way to be tuned at run time. Every deciding literal lives in rules/climate.v1.json, which carries its own citations, so domain judgement enters in exactly one place and you can read it.
Run ./run.sh killswitch and the pipeline runs twice, once with the model layer on and once with it off. It prints two hashes. 93 model-written explanations vanish and zero verdicts change.
The comparison, including where we lose
The track asks for the workflow run against a single-prompt baseline on the same test cases. Both arms read the identical frozen claim set of 97 claims, so an imperfect extractor is unfair to neither.
On the strictest measure, the baseline beats us. Exact five-field match against a blind gold set: baseline 26 of 60, the Gate 23 of 60. There are 14 field-level checks the single prompt gets right and we get wrong — 11 we miss, 3 we over-fire.
What the workflow wins is repeatability. Over three identical runs the Gate is byte-identical all three times, because it is a pure function. The single prompt changed its verdict on 8 of 97 claims between runs, and on 19 of 97 at the field level.
target_year is 60 of 60 against the blind gold set, including a sentence that states a target year and a baseline year together and splits them correctly.
What it gets wrong, and we know
Scope 1/Scope 2/Scope 3are matched as a named standard. Scopes are defined by the GHG Protocol, but a sentence that only says "Scope 1" names no standard. Precision on that field is 37.5%.quantitymisses "more than half", "$1 billion", and two-digit fiscal shorthand likeFY20.- The Gate has no negation handling and no pledge detector. An anti-pledge — "the board has decided not to cut emissions by 50% by 2030" — returns the identical verdict array as the real pledge, because both state a number, a year, a standard and a named body. That is correct for a tool that measures whether a sentence is checkable rather than whether it is honest, and five named tests pin it. A reader still has to notice the word "not".
Every one of these is pinned by a passing test rather than quietly fixed.
An honest note about method
Two annotators produced blind gold labels. They turned out not to be independent — one read the other's committed file before finishing. So there is no inter-annotator agreement figure anywhere in this entry, on any field, even the ones where the two rubrics agreed.
What survives is better sourced: the two labelling rubrics were written before either annotator saw the other's work, and both are in git with timestamps. They defined accountable party oppositely — one required a named role or governance body, the other accepted the company itself — and that single choice moves 29 of 60 claims. Two careful readers disagreeing about what "accountable" means is a real finding about pledge language.
One limit this repository cannot repair: the rubric-before-labels ordering is auditable in git, but "the annotator did not read the results before labelling" is attested, not provable. Stated here rather than left to be found.
Findings
Across the 60 claims taken verbatim from Microsoft, Apple, Nike and Patagonia, not one names an officer or body accountable for the pledge. Eleven of the 37 invented claims do, because we wrote those.
Built with
Python standard library only in the core — re, json, hashlib, dataclasses, pathlib — enforced by a test that fails the build if that list grows. 82 tests. Gemini 3.6 Flash for the model layer until its free tier hit 20 requests a day, then Featherless serving DeepSeek-V3; adding the second provider was one new module and one registry line with the core untouched, which is the migration that proves the boundary was real.
Built with Claude Code driving a multi-agent system: separate agents for research, strategy, architecture, the design of the workflow poster, the build, a judge simulator and a red team that attacked the claims. I directed it and made the calls. The red team found three of the defects listed above, and killed the agreement statistic.
Log in or sign up for Devpost to join the conversation.