Shakedown

Chaos customers, on purpose: test shoppers who pay twice, swap carts and argue with your AI support bot in PayPal's sandbox. A receipt then shows what your checkout would have lost.

Comment

The idea in one minute

When you pay for something online with PayPal, the shop's website and PayPal pass a few messages back and forth. The shop asks PayPal to set up the payment. You approve it. The shop tells PayPal to take the money. PayPal confirms it.

If the shop's code handles one of those steps carelessly, money goes missing. A customer is charged twice. Goods ship that were never paid for. A support chatbot refunds more than the shop allows. I call each of these a leak.

Most shops only test the polite path: one customer, one click, one payment. Real customers aren't polite. They press Pay twice when the page is slow. They change their cart in another tab. Their card gets declined. They argue with the chatbot.

Shakedown is a team of test customers who do exactly those things, on purpose, before real customers do. I call them the chaos customers. Afterwards, Shakedown checks PayPal's own records to see what really happened, and prints a receipt: every leak, what it would cost to the cent, the proof, and how to fix it.

It only ever runs in PayPal's sandbox, where the money is fake, and only against a store you own.

A few words I use:

  • Sandbox: PayPal's test version of itself. It works like the real thing, with pretend money.
  • Capture: the moment PayPal actually takes the customer's money.
  • Webhook: a message PayPal sends to the shop's website when something happens, such as "the payment went through".
  • Seed: a number that makes a test run repeatable: the same seed sends the same customers doing the same things.
  • CI: the automatic checks that run whenever a developer changes the code (here, GitHub Actions).

Inspiration

Before a new ship takes passengers, it goes on a shakedown cruise: a trial voyage to find what breaks while it's still safe to fix. Online shops don't get one. A checkout passes its tests, goes live, and the first sign of a problem is money that's gone.

I wanted a shakedown cruise for payments: a cast of difficult customers you can send through your own PayPal checkout before launch, which tells you exactly what would have leaked.

What it does

Each test customer tries one thing real customers do:

Test customer What they do What Shakedown checks afterwards
The Double-Clicker Presses Pay twice, or retries after the connection drops The customer was charged once, and the order shipped once
The Cart Shuffler Changes the cart after approving the payment, or sends their own price The shop never shipped more than PayPal actually collected
The Echo Sends a "you've been paid" message that PayPal never sent, repeats a real one, or delivers one late The shop acts only on messages PayPal really sent, acts on each one once, and doesn't let a late one undo a newer one
The Bouncer Pays with a card that gets declined Nothing shipped when the payment failed
The Policy Lawyer Chats with the shop's AI support bot and pushes for refunds the policy doesn't allow The bot never refunded beyond the written policy, and never promised a review that nobody will do

How a run works:

  1. You point Shakedown at your own store, in the sandbox.
  2. The test customers go through your checkout, the part of your site that receives PayPal's messages, and your support bot.
  3. Shakedown reads PayPal's records to see what really happened: what was charged and what was refunded, order by order.
  4. Plain code, not AI, compares that with what your store did (what it shipped, refunded and believed), and prints the receipt.
  5. You fix the code and run it again with the same seed. When everything is fixed, the receipt comes back at $0.00.

The receipt looks the same everywhere you see it: in your terminal, as a comment on your GitHub pull requests (so a leak can stop a change from going live), on the website, and in the console.

How it differs from PayPal's own checker. PayPal's AI Toolkit includes /paypal:doctor, which reads your code and points out likely mistakes. Shakedown works on the running store instead: its customers go through your real checkout, and every finding comes with the sandbox order behind it and the amount it cost.

The demo store. To show it working, I built Leaky Llama Supply Co., a pretend shop with common mistakes, one switch per customer. With every mistake switched on (recorded run 74A923AC, October 4, 2026, seed 2026), the four checkout customers found 8 leaks: $467.00 the shop would lose and $24.00 customers were overcharged. With the fixes applied, the same run comes back at $0.00.

Try it in two minutes

  1. Open https://shakedown-web.onrender.com. No sign-in needed.
  2. Press Run the demo shakedown. Four test customers go through Leaky Llama, and the receipt prints one leak at a time, as PayPal confirms each one.
  3. Press Apply fixes and re-run. The same customers come back, and the receipt seals at $0.00.
  4. Open the console at https://shakedown-web.onrender.com/app to see every finding with its proof and its fix.

Where AI helps, and where it doesn't

AI is good at playing a difficult customer. It isn't trusted to be the judge.

  • AI (Claude Haiku 4.5) does the talking: it plays the Policy Lawyer in the support chat, turns a shop's written refund policy into rules, writes a first plain-English explanation of each finding, and powers Triage, the console's assistant.
  • Plain code does the judging: whether money leaked, how much, and whether the CI check fails. It reads PayPal's records, so the same run always gives the same answer.

For example, on the website, the shop's support bot tells a customer "All set" after refunding an order in two parts. PayPal's records show the second refund, $34.00, went past the shop's written rule: the bot may refund at most $100.00 per order on its own.

The AI calls stay cheap: every one goes through a spending cap, and repeat runs reuse saved answers. All the AI calls made while building Shakedown cost $0.5283, over 178 calls (to October 5).

How I built it

  • The engine (packages/core) sends the test customers through a store, reads PayPal's side of every order through the sandbox API, and grades each check in plain code. Every run is repeatable from its seed.
  • The demo store (apps/leaky-llama) is a Next.js shop with a real PayPal sandbox checkout (PayPal's JavaScript SDK v6 card fields), a listener for PayPal's webhooks, and Lulu, an AI support bot built on Claude.
  • The website and console (apps/web) run hosted demo runs on Render Workflows, one task per customer, streaming each step to your browser. The console is built on AG Studio 3, with my own widgets and Triage, a custom agent on its Agent Framework that answers questions about the runs.
  • The command-line tool: npx @shakedown-dev/cli run prints the receipt and fails when anything leaks. In GitHub Actions, the same command posts the receipt on the pull request.
  • Stack: TypeScript, Next.js 16, React 19, Drizzle ORM (PGlite on a laptop, Postgres on Render), Turborepo, Vitest, Playwright with axe, and Biome. The video is made in Remotion from real screen recordings.
  • Built solo, with Claude Code.

PayPal APIs used, and why

Every call goes to PayPal's sandbox. A lock in my PayPal client refuses any other address.

PayPal API What it's for in Shakedown
Orders v2: create, capture and read an order The demo store's checkout. Shakedown reads every order back to see what PayPal really holds.
Payments v2: captures and refunds PayPal's record of money taken and money returned. Every verdict comes from here.
Webhooks: verify a message's signature The demo store uses it to check that a message really came from PayPal. The Echo tests whether a store skips that check.
OAuth 2.0 The access tokens for each sandbox call.
JavaScript SDK v6 card fields The card form on the demo store's checkout page.
PayPal Agent Toolkit The refund tool inside the demo store's AI support bot.

The Double-Clicker's fix is a request ID: a tag on each request that lets PayPal recognise a repeat, so a second click doesn't become a second payment. PayPal's Agent Toolkit has an open issue about exactly this (paypal/agent-toolkit#98): by default, its create and capture calls send no request ID, so an AI agent that retries a slow payment can hit an error after the money has already moved. I reproduced it in the sandbox and sent PayPal a fix, with the package's first tests: paypal/agent-toolkit#108.

On October 29, 2026, an independent study of six payment providers' AI toolkits (double-take) published its PayPal results: a refund that goes through but comes back as a timeout can be made twice, because no request ID is sent. It cites the pull request, which is still open.

Measured results

  • Testing the tester. With each of the demo store's mistakes switched on by itself, Shakedown caught the leak in all 8 leaky cases and raised no false alarm in the 16 correct ones (docs/EVAL-CHECKOUT.md, October 5, PayPal sandbox, no AI involved). Three runs gave identical results.
  • On real code. I tried Shakedown on 22 public projects that take PayPal payments: PayPal's own samples, open-source shops, and apps built on Express, Next.js, NestJS, Hono, SvelteKit, Flask and Laravel. What broke along the way is fixed in the command-line tool's releases 0.2.0 to 0.2.3. In all five live sandbox runs, the store charged the customer twice when Pay was pressed twice: PayPal's standard sample (with its Node and its Python backend), PayPal's v6 sample, an Express tutorial and a NestJS project. The NestJS project also charged €1.00 for a €100.00 booking, because it took the price from the browser.
  • The grader reproduced 32 of 32 known verdicts against scripted support bots (docs/EVAL.md).
  • The policy reader (Haiku 4.5) got 15 of 15 rules right across 3 written refund policies.
  • Lulu, the demo support bot (Haiku 4.5, six refund requests per setup). Given only the Agent Toolkit's refund tool, it correctly refused all 4 refunds the policy didn't allow. But it promised a review it had no way to file, 2 times out of 2: $158.00 of customer harm. With the shop enforcing its policy inside the refund tool, there was none.
  • Quality: 400 automated tests pass. Both apps pass accessibility checks (axe, against WCAG 2.2 A and AA). Lighthouse on a phone scores 95 for performance on the home page and on the console, and 100 for accessibility.

Sponsor tools

  • AG Studio 3 (AG Grid): the console. My widgets (the scoreboard, the cast, where the money would have gone, the finding, every leak in view) sit in AG Studio's layout, alongside Triage, a custom agent on its Agent Framework with a $0.25 spending cap.
  • Render: a Blueprint (render.yaml) sets up the website, the demo store and a Render Postgres database in one go. Render Workflows runs each demo run as one task per customer, with retries. The judges' copy runs with tighter limits (render.judge.yaml).
  • Postman: a 13-request collection for the store's and the console's APIs (docs/postman), validated against collection schema v2.1.
  • APIMatic: its PayPal Context Plugin (acp-paypal) checked my own PayPal client against the PayPal Server SDK's contracts. Its answers led to three fixes: refund request IDs now built on PayPal's capture ID, failed refunds no longer recorded as done, and full responses requested from PayPal (docs/APIMATIC.md).

Challenges I ran into

  • Making the verdict trustworthy. An AI that plays customers is useful. An AI that grades them isn't. So every verdict comes from PayPal's own records, and the same seed always gives the same receipt.
  • Messages only PayPal can sign. A test can't forge PayPal's signature. So against a store that checks signatures, Shakedown can show that the fake messages were refused, but not what the store does with a genuine repeat. There it says inconclusive instead of guessing.
  • Double-clicks are trickier than they look. Pressing Pay twice opens two orders unless each checkout attempt carries its own request ID. And a retried payment ships twice unless shipping also checks for repeats.
  • A $5 AI budget: Claude's smallest model (Haiku) by default, a hard spending cap on every call, and saved answers for repeat runs.
  • Only stores you own: it works only in the sandbox, by design, and a store must prove it's yours with a verification token (docs/THREAT_MODEL.md).

Accomplishments I'm proud of

  • One receipt, the same everywhere: the terminal, the pull request, the website and the console all show the same findings from the same PayPal records.
  • A test of the tester: 8 of 8 leaks caught, and no false alarms in 16 correct cases.
  • A fix for an open issue in PayPal's Agent Toolkit (#98), reproduced and checked in the sandbox, and sent upstream as pull request #108.

What I learned

The costly mistakes aren't in any single call to PayPal. They hide in the gaps between calls: an order created twice, a payment that doesn't match the cart, a message that arrives after the refund. Finding them means behaving like a customer while checking the records like an accountant.

What's next

  • A ready-made GitHub Action around the command-line tool (@shakedown-dev/cli is on npm already).
  • More test customers, and more of PayPal's features: subscriptions, payouts and disputes.
  • Other payment providers, with the same receipt.

Built With

  • ag-grid
  • ag-studio
  • anthropic
  • apimatic
  • claude
  • drizzle
  • next.js
  • paypal
  • paypal-agent-toolkit
  • paypal-orders-v2
  • paypal-payments-v2
  • paypal-webhooks
  • pglite
  • playwright
  • postgresql
  • postman
  • react
  • remotion
  • render
  • render-workflows
  • turborepo
  • typescript
  • vitest
Share this project:

Updates

Submission history