RECEIPTS

Your AI coding agent says "done, all tests pass." RECEIPTS re-runs the evidence in a sandbox and proves whether that's true — VERIFIED, REFUTED, or UNVERIFIABLE, never a guess.

Comment

Inspiration

Every session with a coding agent ends with a confident final message: "Done. All tests pass." "Fixed the race condition." "No other changes needed." Those sentences are the agent grading its own homework — and nothing forces them to be true. I kept hitting the same failure modes: a suite reported green that was run before the last edit; a red suite turned green because the agent quietly deleted the failing test; a "touched three files" claim that actually touched eleven. Catching any of it meant re-reading the diff and re-running the suite myself — re-deriving the exact evidence the agent already claimed to have. RECEIPTS removes that tax.

What it does

RECEIPTS is a post-hoc, evidence-required auditor for completed AI-coding-agent sessions. Point it at a finished session (Codex, Claude Code, OpenHands, OpenCode, Gemini CLI, Copilot CLI, Cline, Amazon Q) and it:

  1. Normalizes the transcript into one provider-agnostic Session Package.
  2. Extracts every claim the agent made — explicit and implied — with a strict schema.
  3. Plans a deterministic verifier per claim and runs it in a locked-down, no-network Docker sandbox.
  4. Builds an Evidence Graph where every verdict traces to the node that produced it.
  5. Adjudicates each claim VERIFIED / REFUTED / UNVERIFIABLE — and marks REFUTED only when a freshly executed command, diff, or file observation contradicts it.
  6. Computes a reproducible Trust Score — one refuted claim caps the score; you can't average away a lie.

It never edits your code, never blocks a merge, and never refutes on a guess. It answers one load-bearing question: did the agent's claims survive a fresh check of reality?

How I built it

RECEIPTS was built with Codex + GPT-5.6 during Build Week, module-by-module from a frozen engineering spec: the verification core, eight session adapters, the sandbox + verifier catalog (pytest execution, an AST-based test-tamper detector, file-stat, scope-check, build/lint), the SQLite Evidence Graph (adjacency tables + recursive CTE traversal + FTS5), the versioned Trust Score, the subsystems (Prompt Studio, Ask RECEIPTS, provider management, dashboard, exports), the plugin SDK, and the React frontend. Architecture decisions are logged as ADRs in DECISIONS.md — the LiteLLM gateway, the SQLite graph, the digest-pinned Docker boundary, and the evidence-required adjudication policy.

GPT-5.6 is load-bearing, not decorative. Whole-session claim extraction runs on gpt-5.6-sol at high reasoning effort — a single long-context pass over the entire transcript, which is what makes cross-timeline stale-success contradictions (a claim asserted early, invalidated by an edit 40k tokens later) catchable at all. Per-claim planning, Ask RECEIPTS, and Prompt Studio run on gpt-5.6-terra, escalating only when the deterministic layer flags complexity.

Codex built it — and Codex is its first audit subject. The first Tier-1 adapter parses Codex CLI rollout-*.jsonl, and the golden fixture corpus is built from Codex-format sessions. The tool Codex built is the tool that checks Codex's homework.

Challenges I ran into

  • Refuting honestly. A false accusation kills an auditor's credibility instantly, so the core invariant is: REFUTED requires attached executed evidence — never a model opinion. The one verifier that could produce a false refutation (a behavior-probe) ships intentionally disabled; its claim class is reported UNVERIFIABLE rather than risk a wrong REFUTED.
  • Transcripts as hostile input. Text inside a transcript that reads like a command ("ignore previous instructions…") is cataloged as a claim, never executed. Verifier commands render only from an allow-listed template catalog; no transcript or repo string is ever interpolated into a shell.
  • Making "why?" one click. Provenance is enforced: if a conclusion can't be traced through the Evidence Graph, RECEIPTS refuses to display it.

Accomplishments I'm proud of

  • REFUTED-precision = 1.000 on the golden fixtures — a single false refutation fails CI.
  • A real, digest-pinned, no-network sandbox — not a mock.
  • Eight working session adapters behind one normalized schema.
  • 114/114 backend tests green; deterministic, reproducible Trust Score.

What I learned

The bottleneck in agentic coding has moved from writing code to trusting it. The valuable primitive isn't another agent — it's an independent checker that re-derives evidence and stays honest about what it can't check. "UNVERIFIABLE" turned out to be a feature, not a weakness.

What's next

Tier-2 adapters (Cursor, Roo Code, Copilot-in-IDE, Aider); enabling the behavior-probe once an eval proves zero false refutations; WASM-isolated third-party plugins; a GitHub Action that attaches a RECEIPTS report to agent PRs. Each is specified at the interface level and recorded in DECISIONS.md — deferred on purpose, not missed.

Built With

Share this project:

Updates