-
-
Add PayPal. Then try to rob it.
-
What Heistproof caught in PayPal's sandbox, and every recorded run.
-
Safe to ship: PayPal refused the first capture, the agent fixed it in one round, then all ten robberies and the AI thief were blocked.
-
A real open-source shop before its maintainer's 2023 fix: a $459.99 item marked paid after PayPal captured $0.01.
-
The AI thief's own robbery: one payment paying for two orders, caught by the books.
-
Find any run by shop, verdict, date or PayPal id.
Add PayPal. Then try to rob it.
Before AI-written checkout code ships, Heistproof tries to rob it for real in PayPal's sandbox. If anything can be stolen, it doesn't ship. Think of a toy shop that sends in a pretend thief before it opens, then counts the piggy bank.
Already caught, in PayPal's sandbox
During the hackathon it found two new problems in real third-party code, one in a widely copied open-source shop and one in PayPal SDK tooling, not in samples we wrote ourselves. Both were reported to their maintainers; the organizers may request the details.
- Real bugs in a real open-source shop. On ProShop v2 (400+ forks), before its 2023 payment fixes, 4 of the 10 robberies got through (see the run).
- A gap the agent's own tests missed. The AI-built checkout passed its 22 tests, yet PayPal refused its first real payment, in 6 of 6 runs. Heistproof caught it and the agent fixed it in one round (see the run).
- An AI thief that finds what lists miss. It found a mistake none of the ten robberies can see, in 3 of 4 live runs (see the run).
Every number and its source: docs/claims.md.
Ready today
- Runs on shops it did not write: a real open-source shop, with its own code untouched.
- Nothing to install to look: every recorded sandbox run is on the hosted console, heistproof-console.onrender.com.
- One step in CI: a shop adds the GitHub Action, and every pull request is checked before it merges, offline and with no secrets.
What it does
- Add. A coding agent adds PayPal, Venmo and Pay Later with APIMatic's PayPal plugin. Or skip it and check a checkout you already have.
- Buy. A scripted shopper pays for real in PayPal's sandbox.
- Rob. Ten scripted robberies, each the test for one PayPal safeguard: a $0.01 price, bad bags, a payment nobody approved, an approval with no payment, a declined card, a double click, a forged webhook, a replayed webhook, a late event, the secret in the browser.
- AI thief. A model sees what any visitor sees and invents robberies of its own. Heistproof carries them out through a few fixed moves aimed only at the shop under test.
- Books. Every order the shop calls paid or shipped must be backed by a completed PayPal capture that covers it. PayPal's records decide, never the model.
- Fix, then ship. What got through goes back to the agent with PayPal's own answer, and everything runs again. It ships only when every robbery fails.
Why it is new
Nobody else runs the whole heist. In one loop, Heistproof buys for real in PayPal's sandbox, runs ten proven robberies plus an AI thief that invents its own against the checkout itself, judges every result by PayPal's records, and sends what got through back to the coding agent until nothing can be taken. docs/prior-art.md compares it row by row with every other entry we found, and says how to refute the claim.
Technological implementation
- PayPal, end to end: JS SDK v6 (
paypal-payments,venmo-payments,paypal-messages,findEligibleMethods); Orders v2 create, capture and get, withPayPal-Request-Id;verify-webhook-signatureand webhook event resends; negative testing withPayPal-Mock-Response; real buyer approvals on PayPal's sandbox pages. It also checks shops whose browser captures the payment itself (JS SDK v5, as in ProShop). - AI that writes, AI that attacks, and a referee that is neither: the coding agent runs headless through the Claude Agent SDK with APIMatic's plugin pinned, on the Meta Model API (
muse-wonder-1.0) in our recordings; any provider's key works. The AI thief is a second model call whose plans Heistproof validates and carries out. - Hard to fool: safe means the real purchase worked, all ten robberies ran and were blocked, nothing the thief tried got through, and the books match PayPal's. Anything less is never safe. Every replayed request first runs unchanged as a control.
- Tested: 92 automated tests on every push (65 unit, 27 robbery tests against running shops in a real browser), run against three checkouts with no PayPal account and no keys.
- Safe to run: the coding agent never holds the model key; a tested guard limits its file and network access; every export is scanned for secrets.
Design
- No keys needed to try it:
npm run demorobs two checkouts in about 90 seconds with an offline PayPal stand-in; GitHub Codespaces opens it in the browser. - A console that replays every run: the verdict, the ten robberies with PayPal's own ids, the AI thief's tries, the books, screenshots. Hosted on Render, nothing to install: heistproof-console.onrender.com.
- One command per need:
demo,check,heist,doctor,setup. Exit codes 0 safe, 1 do not ship, 2 incomplete. - A GitHub Action, offline by default, so every pull request gets robbed with no secrets in CI.
- Any shop: a short
heistproof.jsontells it how to shop; ProShop runs through a small adapter, with its own code untouched.
Potential impact
AI writes checkouts in minutes now, and AI agents are starting to shop and pay (traffic from AI assistants to US retail sites grew 393% in Q1 2026, per Adobe). Checkout bugs don't crash; they quietly let money walk out the door. PayPal already provides every safeguard the robberies test. Heistproof proves, with real transactions, that a shop actually uses them, before it ships and on every change after. It has already found real money bugs in a widely copied open-source shop.
Challenges we ran into
- Shops differ: some capture on the server, some in the browser. Heistproof learns each shop's routes by watching its real purchase.
- A refused capture that the agent's own tests could not see. Only a real purchase shows it.
- Sandbox webhooks can be slow or missing. Heistproof asks PayPal to resend them and never calls an unfinished check safe.
- The AI thief invents different robberies every run. Its plans are kept so results replay in CI, and only PayPal's records grade it.
Accomplishments that we're proud of
- Two new problems found in real third-party code, both reported to their maintainers.
- A capture gap the agent's own tests missed, caught in 6 of 6 full runs.
- An AI thief that found a mistake no scripted robbery can see, in 3 of 4 live runs.
- Every number traceable:
docs/claims.mdlists each claim with its source.
What we learned
Grounded context makes an agent's PayPal code precise; only real transactions make it trustworthy; and a model's own claims are never evidence. The books are.
What's next
Refunds, disputes, subscriptions and agent-initiated checkouts; more shops and stacks; a hosted check agencies can attach to every client handover.
Built With
- apimatic
- apimatic-context-plugin
- claude-agent-sdk
- claude-code
- cloudflare-tunnel
- css
- express.js
- github-actions
- html
- javascript
- meta-model-api
- mongodb
- node.js
- pay-later
- paypal
- paypal-js-sdk-v6
- paypal-orders-v2
- paypal-webhooks
- playwright
- render
- server-sent-events
- sqlite
- typescript
- venmo