Receipts

"Your coding agent says it's done. Receipts checks if the repository agrees."

Comment

Inspiration

1am. Cursor already on the merge button.

The agent's summary said checkout tests passed. Clean run, no red anywhere, npm test sitting right there like proof. I almost clicked. Something made me scroll down one more time first.

A test had been skipped. An assertion was just gone. The command hadn't lied — it really did run successfully. It just didn't run the thing I thought it was checking.

I sat there for a second, not really thinking about the bug. Thinking about how many times I hadn't scrolled down.

What it does

Here's what that moment looks like once it's actually caught.

CLAIM: "Checkout tests pass: npm test" VERDICT: FIX Evidence: test skipped · assertion removed

Same scenario. Green command, quietly broken underneath. The page just states it plainly — claim not supported, do not merge, fix required.

Receipts isn't CI, and it isn't checking the same thing CI checks. CI verifies whether a configured workflow passes. Receipts verifies whether an agent's prose claim about that workflow is actually true. You can have a fully green CI run and a false claim sitting right next to it — that's exactly the gap this closes.

Every verdict is retained in a local Ledger, and each one can be copied as a shareable link to paste into a review thread. Right now that link resolves against the same local evidence server that generated it — genuine cross-machine sharing is the natural next step once this runs somewhere persistent.

What we tried first

The first idea was a trust score. Rate how reliable an agent's summaries tend to be over time. Sounded reasonable for about an hour, until we noticed the problem — a score is still just something you have to believe. We'd built a second thing to trust blindly, dressed up as a number instead of a sentence.

Second idea: have another model review the code and double-check the first agent's work. Also felt smart, briefly. Then it landed — we'd just added a second storyteller to a problem that already had one too many.

The actual shift was smaller than either of those ideas, and took longer to see than it should have: stop asking any model whether something is true. Ask it what's being claimed. Then get it out of the room entirely. Let the repository answer instead — did the test run, is the assertion still there, did anything touch a file it shouldn't have.

How it works

GPT-5.6, through the authenticated Codex CLI, reads the agent's claim and figures out exactly what's being asserted. Extraction runs in a read-only sandbox, prompted not to inspect files or use tools — read-only prevents writes, and the pipeline never gives the model a second call once it returns the claim. Everything after that point is deterministic — the actual commands get re-run, the actual diff gets pulled, tests get checked for weakening, sensitive paths get flagged. The model interprets language. The repository is the only thing that gets a vote on whether it's true.

Four outcomes come out the other side:

  • MERGE — the claim checks out.
  • FIX — the command was green. The evidence wasn't.
  • ESCALATE — the claim is true, but something in the blast radius says a human should look anyway.
  • RE-RUN — the evidence available can't support a determination either way.

This is more trustworthy than asking another model for a second opinion, because it isn't an opinion. It's the same repository the first agent was supposedly working in, answering for itself.

Honest engineering challenges

The hard part was never the model call. It was proving, not just claiming, that the model actually gets removed from the decision. We built the pipeline so nothing downstream can quietly call the model again for a second opinion — that boundary had to hold in the code, not just in the pitch.

The most uncomfortable moment was catching ourselves breaking our own rule. An early version of the Claim Patterns page showed a percentage computed from four data points. We're keeping it to plain counts — "3 of 4 tracked test claims needed a fix" — until there's enough data to say a real percentage means something.

The second uncomfortable moment came later, after we'd already called the receipt-link feature done. A "Copy receipt link" button showed a success message every time, but the actual clipboard wasn't always changing, and a fresh tab with that link sometimes loaded a blank screen instead of the receipt. We'd shipped a UI that claimed success it hadn't verified — the exact failure mode this product exists to catch, just quieter and closer to home. We fixed it to only confirm success after actually reading back the clipboard, with a visible fallback link when it can't, and we made sure a fresh tab now genuinely loads the saved receipt before calling it done.

What we built, and where Codex helped

Codex built most of this alongside us, not just inside it. It scaffolded the pipeline, wrote the fixture-based test suite, and built out the Ledger and Claim Patterns pages, all verified against the full test suite and a production build before we called anything finished. The one moment that stuck with us: when we asked it to simplify Claim Patterns into plain stated facts instead of charts, it noticed on its own that our sample size didn't support the percentage we were showing — and told us before we shipped it. The tool caught the exact mistake it exists to prevent, in its own output, without being asked to check.

What's next

More kinds of claims — performance regressions, dependency drift, migration risk. Running automatically the moment an agent opens a pull request, instead of waiting for someone to remember to ask. And as the Ledger grows past a handful of entries, the patterns that are honestly too small to mean anything yet will finally have enough evidence behind them to say something real.

This was never really about catching agents in a lie. It's about the version of me with my cursor already on the button. Receipts doesn't replace that judgment. It just makes sure that when I trust something, there's finally a reason to.

Built With

Share this project:

Updates