Checkout Proof

Checkout Proof lets AI upgrade old PayPal checkouts, then tests them in the sandbox until real money bugs are gone.

Comment

Inspiration

I went looking for PayPal code in open-source shops, and it was old. In the week to 16 October, 39% of downloads of PayPal's Node server SDKs were of deprecated packages (paypal-rest-sdk, @paypal/checkout-server-sdk), and 222 popular public repos loaded JS SDK v5. I found none loading v6.

Upgrading is exactly the kind of work an AI agent does well now, and PayPal ships the tools for it: the AI Toolkit and the APIMatic server-SDK context plugin. But a migration guide tells you what to change. Nothing runs your checkout afterwards to show that the new one still takes money correctly. And checkout code fails in quiet ways: the order looks paid, the money says otherwise.

The first real shop I tested made the point for me. In expressCart, an open-source Node shop with about 2.2k GitHub stars, a buyer starts a €480 checkout, cancels at PayPal, removes most of the cart and pays the €85 that is left. The shop marks the €480 order Paid, with all 8 items in it.

What it does

Checkout Proof takes a real shop through three steps:

  1. Scan. Static rules find JS SDK v5 (still supported, with v6 available), the pieces PayPal has deprecated, and patterns that lose money (a capture in the browser, an amount check without the currency, no look at the capture's status), each with file and line. Anyone can scan a public GitHub repo on the site.
  2. Upgrade. Claude moves the shop to JS SDK v6 and @paypal/paypal-server-sdk, guided by PayPal's AI Toolkit and the APIMatic context plugin. It never sees credentials.
  3. Prove, and repair. The shop's own server runs against the PayPal sandbox: a real sandbox buyer approving on PayPal, PayPal-Mock-Response declines and refusals, a lost capture response, tampered orders, two tabs paying for one order, a shop restart in the middle of a payment, and PayPal's own verified webhooks. Money rules compare the shop's record with PayPal's (GET /v2/checkout/orders/{id}). Anything that fails goes back to Claude with the evidence, and the proof runs again until every scenario passes.

Plain, deterministic code does the scan, drives the buyer, scripts the failures and decides every verdict. Claude writes the upgrade and the fixes. No model judges its own work.

On two real open-source shops, the proof found 9 money bugs in the original code (16 failing runs); none fails after the upgrade. expressCart passes 15 of 15 scenarios; simple-clothing-store passes 35 of 35 across its demo and production modes (one more, the webhook scenario in production mode, is not applicable: PayPal delivers webhooks only to the hosted site, which runs demo mode). Every verdict on the site carries PayPal's own record: the order and capture IDs wherever an order exists, and debug IDs wherever PayPal answered.

Asked plainly to upgrade, the AI shipped money bugs, and the proof caught them. I upgraded each shop twice: once with my money rules in the prompt, once with a plain upgrade request and no money rule. On simple-clothing-store the plain upgrade failed 6 of 22 sandbox runs at its first proof. It marked a €62.00 order paid for 62.00 MXN, and for €1.00, from PayPal orders that were not its own, and it charged one buyer twice from two tabs. The currency and two-tab bugs were already in the original shop; the €1.00 one was new (the original checks the amount). The failing evidence went back to Claude with no rules added. After one fix round (14 turns, 85 s): 21 pass, 1 inconclusive (my sandbox buyer timing out, not the shop), 0 fail.

On 19 October I added harder scenarios (shop restarts, refused captures, PayPal's payer-action step, a wrong amount or currency on the shop's own order) and never fed their failures back to the plain upgrades. They still fail: simple-clothing-store's in 8 of 34 sandbox runs (4 scenarios, in both modes: it confirmed an order whose capture PayPal refused, ignored PAYER_ACTION_REQUIRED, and lost payments in progress when the shop restarted), and expressCart's plain upgrade, green on the first 13 scenarios, in 2 of 15: it marked a €73.00 order paid when PayPal's answer (simulated by the harness) said 73.00 MXN or €1.00. The upgrades with money rules pass all of them: expressCart's at its first proof, simple-clothing-store's after one fix round from the evidence (42 turns, 221 s).

The proof also caught a double charge in the upgrade with money rules. That upgrade passed at its first proof. Later I edited it by hand to handle PAYER_ACTION_REQUIRED, and the next proof found the buyer charged twice for one order from two tabs: PayPal showed two completed €62.00 captures (0TG628623J5721737 and 3N277234VH1857443). Claude got that evidence back and fixed it in one round (27 turns, 246 s): when a capture's outcome is unknown, it now retries with the same PayPal-Request-Id and blocks other PayPal orders for that shop order until it knows. The next proof passed 28 of 28. (I later reverted my hand edit and had the fix loop redo it, so every change in the shipped upgrade is Claude's; the double-charge fix stayed.)

Judges can run it on the live site, with the code in the testing instructions. "Re-run the proof" replays every scenario against the upgraded shop, with fresh sandbox orders. "Watch the original lose money" runs the original shop in two scenarios, and comes back red in about a minute: one order paid twice, and 62 MXN accepted for a €62 order. The harness refunds those test captures afterwards. Every live run since 19 October 08:00 UTC is saved whole and listed on the page.

How I built it

  • The harness is TypeScript on Node 22. Each shop gets an adapter that starts it, reads an order from the shop's own record (MongoDB for expressCart) and drives its checkout. Playwright plays the sandbox buyer through PayPal's real approval pages.
  • Failures are scripted at the PayPal boundary. A Node preload in the shop's process adds PayPal-Mock-Response to the shop's next PayPal call (INSTRUMENT_DECLINED, INTERNAL_SERVER_ERROR, PAYER_ACTION_REQUIRED, TRANSACTION_REFUSED, COMPLIANCE_VIOLATION, MAX_NUMBER_OF_PAYMENT_ATTEMPTS_EXCEEDED), drops a response after PayPal has captured, or (labelled SIMULATED on every page) rewrites a capture to PENDING, and logs every PayPal call.
  • The money rules are plain code: a shop order counts as paid only if PayPal holds exactly one completed capture for it, in the right amount and currency, for a PayPal order that shop created. Each verdict lists its reasons, the shop's HTTP answers, PayPal's record and debug IDs.
  • The upgrade and fix loop runs claude -p (claude-opus-5-5) on a work copy of the shop, with PayPal's AI Toolkit plugin and the APIMatic context plugin loaded, and no credentials in its environment. A failing proof is turned into a fix prompt (verdict reasons, shop responses, PayPal's record), the model commits a fix, and the proof runs again. Rounds lost to my own harness problems are kept in the log and greyed out on the site, never counted.
  • Mutation tests answer "can the proof fail?": the harness removes one fix on purpose (the amount and currency check, the refusal handling, the saved payment state, PayPal's own payer-action link) and the matching scenarios must turn red. They do.
  • Webhooks: the upgraded simple-clothing-store got a second model pass that adds a verified PayPal webhook route. On the live site, a capture the harness presents to the shop as PENDING (simulated) is confirmed by PayPal's genuine PAYMENT.CAPTURE.COMPLETED delivery, refunded through a genuine PAYMENT.CAPTURE.REFUNDED, and replayed and forged deliveries change nothing.
  • The site is React + Vite, served by a plain Node server on Google Cloud Run, with Firestore for daily caps and saved live runs and Secret Manager for the sandbox credentials. Every number on it is generated from the run records, and a test fails if the README drifts from them. Everything runs in the PayPal sandbox; no real money moves.

Challenges I ran into

  • Making a test that can fail, and saying so. It is easy to build a proof that always comes back green. I added the plain-prompt control, mutation tests, and a live run of the original shop that is supposed to come back red.
  • Telling the shop's failures from mine. A sandbox buyer timing out, a port left open by a killed run, a slow webhook: early on, those showed up as shop failures, and once the model spent a fix round chasing one. Now a verdict is FAIL only when the shop is at fault; harness and PayPal-side problems are INCONCLUSIVE and say why.
  • Keeping the AI's part honest. At one point I made a change by hand inside the proven chain. I reverted it and let the fix loop write it instead, and the site labels every step as Claude's, a hand edit, or the harness's.
  • Things the sandbox won't do on demand. There is no switch for a real PENDING capture or a real PAYER_ACTION_REQUIRED, so those use PayPal's mock header or a labelled simulation, and the site says which.
  • Running live proofs for judges at an expected €0. Cloud Run gives CPU only while a request is open, so a run holds a request to its own URL for as long as it lasts, open to the judges only, behind daily and monthly caps and a hard €2 billing cap that switches the demo off.

Accomplishments that I'm proud of

  • A receipt that explains the problem in one look: the buyer paid €85; the shop marked a €480 order paid.
  • The plain-prompt result: the proof caught real money bugs in a straightforward AI migration, and Claude fixed them from the evidence alone.
  • 9 bugs found in two real open-source shops, all gone after the upgrade: 6 shown against PayPal's real record, one a crash, one against a simulated pending capture, one that rests on an assumed Printful reply (each labelled on the site).
  • A live button that makes the original shop lose money on demand, in about a minute, with fresh sandbox IDs.

What I learned

  • "Migrated" and "takes money correctly" are different claims. Only running the checkout against PayPal tells them apart: asked plainly to upgrade, with no money rule, Claude kept two of the old shop's money bugs in clean, modern code and added a third (it stopped checking the amount).
  • The most valuable thing to hand an AI is evidence, not rules. A failing scenario with PayPal's record attached got a correct fix in one round, without new instructions.
  • Order status is not payment. The bugs I found come from trusting something other than PayPal's capture: an amount without its currency, an order the buyer never approved, or a session that outlived its cart.
  • PayPal-Request-Id is the single most important parameter for a safe capture: it is what made the double-charge fix work.

What's next for Checkout Proof

  • Card fields and card declines (JS SDK v6 card fields), where real 3-D Secure lives; refunds through the shop; authorize-then-capture.
  • Let the model write the per-shop adapter too, so a new shop needs no hand-written code, and run it on a third shop.
  • A GitHub Action that runs the proof on every pull request that touches checkout code, and posts PayPal's record with the verdict.

Built With

Share this project:

Updates

Submission history