Receipts

• 

  Every pull request makes a claim. Receipts writes the missing test from the issue and shows the evidence.

• 

  A green CI check isn't proof: the same pull request with and without a blind test written from the issue.

• 

  199 SWE-bench Verified patches: Receipts passed 10% of wrong patches; a model reading the diff passed 46%.

• 

  Same xarray issue: the real fix is Proven (fails 3/3 before, passes 3/3 after); a wrong patch is Refuted.

• 

  The verdict lands on the pull request as a GitHub check, with a link to the full receipt.

• 

  No-sign-in demo: check a real fix, an empty patch or a wrong patch and watch the receipt print.

Inspiration

A green CI check says the existing tests still pass. It does not show that a pull request fixes the bug it claims to fix. Maintainers now review a growing share of AI-written pull requests, and reading the diff is not enough: in our evaluation, Nemotron Ultra reading the issue and the diff accepted 46% of wrong patches.

What it does

Receipts writes the missing test from the issue and the original code, without showing the test writer the proposed patch. It runs that test three times on the original code and three times with the pull request, then runs the existing tests. The result is a receipt: the claim, the generated test, command output, every run and a verdict (Proven, Refuted, Regression, Unproven or No checkable claim). Reviewers inspect evidence instead of a model's opinion of the diff.

On 199 patches across 40 SWE-bench Verified issues, Receipts accepted 10% of wrong patches. Nemotron Ultra reading the issue and diff accepted 46%.

It is built for maintainers and reviewers who need to know whether a change does what its description says before they merge it.

How we built it

Every model call runs through Nebius Token Factory, and each Nemotron size has one job:

• Token Factory Sandboxes set up the repository and run pytest. Forking a built sandbox lets the same accepted test run on separate original-code and pull-request copies.
• Tavily adds short library-documentation notes when the issue names an API. Code hosts are excluded and the writer gets no links, so the issue still defines correct behavior. With no API names, research is skipped; a failed search is recorded and the check continues.
• The verdict is deterministic. Execution results decide it. Before Refuted, Ultra reviews whether the test matches the issue, and any doubt keeps the verdict Unproven.
• A GitHub App puts checks on pull requests, and shareable receipts keep the evidence.
• React and TypeScript run on Vercel, the FastAPI API on Render, Neon stores sessions and saved runs, and LangSmith traces the pipeline. Model inference and test execution run on Nebius.

A live check

PR #14 in a booking API: "Reject bookings that end before they start." Receipts returned Proven in 42 seconds, using 68K tokens. The generated tests cover end-before-start and zero-length bookings, both expecting HTTP 400. The original code returned HTTP 201, so both tests failed in all three runs; with the PR, both passed in all three. All 13 existing tests still passed.

Results and limits

Execution-backed evidence trades coverage for strength: confirmed real fixes fell from 96% to 66%, and some checks stayed Unproven or rejected real fixes. These are sample results, not a correctness guarantee. The methodology and limitations are public.

Challenges and what we learned

The hardest problems were keeping generated tests faithful to the issue, telling infrastructure failures apart from meaningful test failures, and avoiding overconfident negative verdicts. Scope checks, repeated runs, isolated sandbox copies and second opinions address each one. We learned that reliable coding agents need budgeted model roles, execution-based checks and explicit uncertainty, not just confident answers.

What's next

• Raise real-fix confirmation above 66% without accepting more wrong patches.
• Support test frameworks beyond pytest.

Try it

Open the no-sign-in demo, or sign in with GitHub to check your own repositories (5 checks a day per account). The backend and agent and the frontend are open source under Apache-2.0, with setup instructions.

Built With

• deepagents
• fastapi
• github
• langchain
• langsmith
• nebius
• neon
• nvidia-nemotron
• postgresql
• pytest
• python
• react
• render
• tavily
• typescript
• vercel