The Promise Ledger

A machine-verified ledger of public product promises. It extracts a dated, falsifiable promise, then checks — with zero LLM — whether the official page, as archived on the deadline, shows it shipped.

Comment

Inspiration

Every week a tech company makes a dated commitment: "open weights by the end of the year," "generally available next quarter," "available later this month." Some ship. Some ship late. Some quietly never happen. There is no honest, checkable scoreboard — only press releases and vibes. We wanted a ledger where the outcome of a promise is decided by a rule you can read, against a page anyone can open, with no model getting a vote.

What it does

The Promise Ledger has already caught two AI companies quietly dropping a dated public commitment — Anthropic's "Claude 3.5 Haiku and Claude 3.5 Opus later this year" (Opus 3.5 was never released and the name left the roadmap) and Apple's personalized Siri — and it marks each ABANDONED with the original quote still attached.

It turns an announcement into an accountability record in four steps:

  1. Extract — a Gemini agent (Google ADK, structured output) pulls one falsifiable promise out of the text: the verbatim source quote, a normalized deadline, an observable outcome, and machine-checkable keywords.
  2. Audit — a second agent on Gemma (a genuinely different model family) adversarially checks whether the promise is actually checkable. A rejection sends the extractor back with a specific instruction, up to twice.
  3. Gate — pure Python, no LLM: a real calendar deadline that isn't absurd, at least two distinct specific keywords, a substantive outcome. The gate admits the promise — the model never does. The announcement text is untrusted and flows straight into the extractor's prompt, but a hostile announcement can't talk its way to a verdict: FULFILLED / ABANDONED / … is not a field any LLM in this system can write — only the gate and the zero-LLM verifier set a status.
  4. Verify — zero-LLM, point-in-time. Two probes: (a) the official evidence page as archived by the Wayback Machine on or before the deadline — if the check keywords are in that capture, the promise was kept on time and the capture timestamp is the dated proof, no prose-date guessing and no source to trust but a neutral public archive; (b) the page now, which combined with (a) separates late from delayed from abandoned. A fixed decision table produces one of seven statuses (PENDING, FULFILLED, FULFILLED_LATE, PARTIALLY_FULFILLED, DELAYED, ABANDONED, UNVERIFIABLE), and every verdict records how it was reached (wayback@deadline, live-page, …).

A per-company and overall scorecard rolls this up into an on-time rate that is a count, not a claim. When delivery is proven but nothing pins a date, the promise is FULFILLED (undated) and held out of the rate — neither on time nor late. The whole ledger is exposed over the Model Context Protocol — get_scorecard, list_promises, get_promise, admit_promise, run_verification_cycle — so an agent, the web UI, or a CI job all read and write the same verified record.

The demo ships seeded with 14 real promises across 7 companies (Anthropic, Apple, Google, Meta, OpenAI, Stability AI, xAI), each quote transcribed from the linked primary source. Several verify point-in-time against an official page as archived on or before the deadline — the Meta Llama 3 and Llama 3.1 posts, Anthropic's Claude 3 Haiku page — where the capture timestamp itself is the dated proof. Outcomes are kept visible, not smoothed: 2 ABANDONED (Anthropic's Claude 3.5 Opus and Apple's personalized Siri — no delivery evidence 180+ days past the deadline), 1 FULFILLED_LATE (Anthropic's Claude 3.5 Haiku, four days into November after a "later this month" promise), 3 FULFILLED (undated) (delivery proven, nothing pins the date). OpenAI's ChatGPT-for-Windows page blocks non-browser clients, so it can land UNVERIFIABLE on a slow pass — and the 6-hourly cycle recovered it to FULFILLED from its 2024-12-28 capture on the next run, which is the self-healing working. The seeded on-time rate sits around 72% and moves on its own every cycle — that's the design, not a bug.

How we built it

  • Google ADK LlmAgent + InMemoryRunner for the extractor (Gemini 3.5 Flash-Lite) and the auditor (Gemma). Structured Pydantic output schemas on both.
  • FastMCP server exposing the ledger over MCP (streamable-HTTP).
  • FastAPI backend: a Server-Sent-Events endpoint streams every pipeline stage as its real agent result arrives, plus read endpoints for the scorecard. A single self-contained static page renders it.
  • The verifier is plain Python + httpx: the Wayback availability API for the point-in-time capture, then fetch → strip scripts/markup/Wayback-toolbar → decode entities → whole-token keyword match (word-boundary aware so "GA" doesn't match "navigation") → fixed decision table. Prose-date parsing is a fallback for when no capture near the deadline exists.
  • Storage behind a small backend protocol: a locked JSON file (the seeded demo ships this way, no cloud credentials needed), Firestore, or in-memory for tests.
  • Google Cloud Run for the public web service, and a separate Cloud Run Job (ledger-cycle) that runs python -m ledger.run_cycle --all every 6 hours on Cloud Scheduler — the zero-LLM re-verification pass. No API key, no model call. The Job and the web service share one Firestore ledger, so a promise coming due, an official page changing after the fact, or a transient fetch failure clearing up is picked up on the next pass and the public scorecard moves with no human in the loop.
  • 85 tests, no network, no LLM. An adversarial QA pass (12 findings) plus a judge-style audit hardened the verifier, the gate, the scorecard, XSS in the UI, and the MCP surface.

Challenges we ran into

  • An official page only shows its state today. A changelog is a rolling window; a pricing page shows this week's prices. So "was it true by the deadline" often can't be read off the live page at all. The fix is the Wayback Machine: fetch the official page as captured near the deadline, and let the capture timestamp be the dated evidence. That turned a hand-picked-URL problem into a deterministic, tamper-evident one.
  • The archive isn't complete, and some vendor pages are anti-bot / JS shells. When there's no usable capture on/before the deadline, the verifier falls back to a date read off the current page — a weaker signal, biased toward FULFILLED over FULFILLED_LATE. A few rows (xAI Grok-2, Apple ChatGPT-in-iOS, Google Gemini Advanced) point at the relevant Wikipedia article rather than the vendor page, because the vendor page renders as an empty JS shell to a non-browser client; Wikipedia is neutral, dated and archived, and every such row carries a comment saying so in seed_data.py. Anthropic's Claude 3.5 Haiku row still points at a third-party dated write-up because its official changelog rolled over with no useful pre-deadline capture.
  • Keyword brittleness. If a page describes a shipped feature in words that don't contain the check keywords, the verifier under-reports it — it stays FULFILLED (undated) rather than being assumed on time.
  • Bot blocks, routed around. Some official pages 403 non-browser clients; the verifier retries against a recent archived capture before giving up as UNVERIFIABLE.
  • Not letting the model decide. Every outcome stays in deterministic code — the LLM only proposes structure, it never grades.

What we learned

An "AI accountability" tool is only trustworthy if the AI isn't the judge. Pushing every verdict into a fixed, public rule made the product both defensible and genuinely useful — and forced us to be honest about the cases the rule can't settle.

What's next

The Promise Ledger is one of four projects built on the same spine — an LLM proposes structure, deterministic code decides the outcome, and the evidence comes from an independent source: the Wayback Machine here, a government shell-company blacklist in Trusted Hire México, versioned business rules in ScopeCouncil, a cited linguistic archive in Language Recovery OS.

  • A larger, continuously growing promise set across more sectors (not just AI roadmaps).
  • Corroborate against immutable registries too — HuggingFace / GitHub Releases APIs give a dated, machine-readable "when did X ship" with no scraping at all.
  • An append-only, hash-chained ledger so a past verdict can't be quietly changed.
  • A public, embeddable per-company badge driven by the live scorecard.

Built With

  • cloud-scheduler
  • fastapi
  • fastmcp
  • firestore
  • gemini
  • gemma
  • google-adk
  • google-cloud-run
  • httpx
  • model-context-protocol
  • pydantic
  • python
  • wayback-machine
Share this project:

Updates