Inspiration
Reports written by AI have started failing in public, and not quietly. Deloitte Australia refunded part of a government contract after an AI written report cited research that does not exist. EY withdrew a study after researchers found hallucinations in it. KPMG pulled an AI report after only 5 of its 45 citations matched real sources. The failure was the same every time: confident text with nothing behind it. The obvious fix, asking the AI to check itself, does not work because it grades its own homework. I wanted something that could not mark its own exam.
What it does
Receipts takes a company briefing written by AI, pulls out the claims that can be checked, numbers and dates first, and checks each one against live web search. The model may only propose a source and a quote. A referee built in plain code then decides. A claim is kept only if the source is one of the search's own results, the quote really appears in that source, the claim's numbers and dates appear in the quote, and the quote is not a rumor, a plan, or a denial. The referee can lower a verdict but never raise one. What survives is rewritten into a verified briefing with footnotes. Everything else is struck through, labelled as a plan, or handed back as "needs your check." Nothing is silently guessed. In Attack mode you can type any single claim and run it through the same pipeline live.
How we built it
built the backend first and the UI last, which is the opposite of how it looks. On Saturday evening the referee rules, R0 through R9, and the Linkup search wrapper went in before a single line of frontend existed, because the rules are the actual product. Then came the agent loop: plan, search, judge, referee, retry. After that came the Anthropic wrappers and a real evaluation harness. We ran it against 4 real companies before the UI was even sketched, so the numbers could not be flattered by cherry picked demo inputs. Only then did we add the FastAPI app, the plain HTML, CSS, and JavaScript frontend with no build step and a strict CSP, the rewrite step, and the hand labelling page so the evaluation could be scored honestly later. On Sunday we restyled the whole UI to match a Figma Make exploration, hand labelled all 32 pinned claims, scored them, and recorded a demo straight from the running app.
Challenges we ran into
The referee started too strict. At an early checkpoint the judge proposed "supported" 43 times and the referee kept only 13, mostly because the word overlap and number checks were pickier than they needed to be. We fixed the worst of it, and the hand labels later confirmed exactly how much recall that cost, 41 percent, rather than hiding it.
Some bugs were almost funny. The model once flagged "May" the month as the hedge word "may." A file rewrite partway through corrupted the source encoding, so status icons rendered as garbage characters (âœ") until a real headless browser test caught it. The same test caught every result card being drawn twice.
The hardest problem was not code. It was honesty. Once all 32 hand labels came back true, the evaluation could not measure a false positive rate at all because there was no false claim in the sample to catch. It would have been easy to quietly drop that caveat. We did not.
Accomplishments that we're proud of
A safety check that is actually deterministic Python, not a prompt asking the model to behave. The model's own stated reason is shown in the UI and explicitly labelled as not trusted on its own. A real evaluation before the demo existed: 32 hand labelled claims, scored by a script anyone can run again, with the result reported even when it is not flattering, 41 percent recall and precision unmeasured. 88 automated tests plus a real headless browser end to end test that drives the actual running app. Every capability claim in the README points at the file and line that makes it true.
What we learned
A referee that can only lower a verdict is much easier to trust than one that can also raise it, and much easier to explain to a judge in one sentence. Hand labels are only as useful as their spread. All true labels measure recall for free but tell you nothing about precision, and saying that plainly is worth more than a polished number that overclaims. A real browser test also catches things a human skim never will, like a garbled icon or a doubled card, faster than redesigning anything.
What's next for Receipts
An adversarial benchmark with planted fake quotes and changed numbers, to finally measure the false positive rate that the current label set cannot touch. A "Report citations" mode, already designed in Figma, that checks a report's reference list the same way, title, author, and year, instead of only its prose claims. Raising the per run claim limit past 8 now that the budget and caching make a longer run affordable. Tightening R2 to check units, not just scale suffixes, so "$8 billion in convertible notes" and "$18 billion in total" are never conflated again.
Built With
- anthropic
- claude
- claude-haiku
- claude-sonnet
- css3
- elevenlabs
- fastapi
- figma
- figma-make
- html5
- hyperframes
- javascript
- linkup
- node.js
- pydantic
- pytest
- python
- restapi
- server-sent-events
- uvicorn
