Inspiration

Payment bugs are usually found by customers. PayPal delivers webhooks at least once, so the same event can arrive twice. It doesn't promise any order, so a refund can arrive before the payment it refunds. A capture can sit in PENDING. Most integrations are tested by clicking through checkout once in the sandbox, the happy path, and then shipped. The rare paths break later: an order ships twice, a refunded order still shows as paid, a forged webhook gets accepted.

PayPal gives developers the parts: the sandbox, a webhook simulator that sends one static sample event, and the PayPal-Mock-Response header for errors. Nothing ties them together into a test that runs the whole situation against your app and checks the result.

What it does

Rehearsal plays real PayPal payment situations against your app in the PayPal sandbox and grades what your app believes after every step.

  • Pick a scenario, or describe one in plain English. For example: "A buyer pays $9 for ginger chai, PayPal sends the completed webhook three times, and the shop must ship only once." Gemini turns this into a test plan. Code validates every step, amount and event type before anything runs, and invalid drafts go back to the model with the errors.
  • It runs the plan for real:
    • Orders v2 create and capture with sandbox cards.
    • Payments v2 refunds.
    • Negative testing through PayPal-Mock-Response (declines, pending captures).
  • It replays PayPal's real signed webhooks. Each delivery is verified with verify-webhook-signature, matched to its run, and forwarded to your app as sent, twice, late, out of order, or with a tampered signature. If the sandbox doesn't emit an event, Rehearsal builds a clearly labelled synthetic one from the real sandbox resource.
  • It grades your app's state after each step, with a full timeline of PayPal calls, webhooks and app responses in AG Grid.
  • When a step fails, an AI diagnoser explains it. Gemini reads the plan, the timeline and the handler code, and returns the cause, the fix, a patch, and the step that proves it.

The demo shop, Chai Corner, comes in a buggy and a fixed version. The buggy one has five mistakes real integrations ship with:

  • It treats an approved checkout as paid.
  • It ships again when the same webhook is redelivered.
  • It treats a PENDING capture as paid.
  • It never verifies webhook signatures.
  • It ignores refunds.

Live results (10 Nov 2026, PayPal sandbox)

Eight scenarios ran against both shops on the live deployment:

  • Buggy shop: fails exactly the five scenarios that match its five bugs.
  • Fixed shop: passes all eight.

The bug it caught in our own code. On its very first live run, Rehearsal failed our fixed shop on "events out of order". PayPal sent COMPLETED then REFUNDED. Rehearsal delivered them reversed, and the late COMPLETED re-opened the refunded order and shipped it. Our unit tests had never tried that order. The AI diagnoser pointed at the COMPLETED handler and proposed the same one-line fix we shipped. We added a test from that run, redeployed, and the live re-run passed.

How we built it

  • Node.js and Express, deployed on Google Cloud Run.
  • PayPal: Orders v2, Payments v2, Webhooks (registration and verify-webhook-signature), and negative testing headers, all in the sandbox.
  • A webhook relay that is the URL registered with PayPal. It matches each event to its run and re-delivers it in the mode the scenario asks for.
  • A plan schema in zod with a strict step whitelist (create_order, pay, refund, deliver, deliver_out_of_order, expect, wait), and money and currency checks in code.
  • Gemini (gemini-3.8-flash) for two jobs: compiling English into a plan, with a repair loop, and diagnosing failures. The model only proposes; code decides what runs.
  • AG Grid for the run timeline.

Challenges

  • The sandbox's own limits. Our first sandbox business account was registered in India, and it can't take card payments (PAYEE_NOT_ENABLED_FOR_CARD_PAYMENTS). We moved to a US sandbox business account.
  • Missing events. Some events never arrive from the sandbox, or arrive late. Rehearsal waits for the real event first, then falls back to a labelled synthetic one, so a run never silently passes.
  • Keeping the AI honest. The compiler can only produce whitelisted steps, every amount is checked in code, and the diagnoser must cite the step that proves its claim.

What we learned

Running scenarios against a live sandbox finds bugs that unit tests miss, because unit tests only try the orders we think of. Our own "fixed" shop proved it on the first live run.

What's next

  • A CLI and a GitHub Action, so every pull request rehearses the payment paths.
  • More scenarios: disputes, subscriptions, partial refunds.
  • A drop-in adapter for common stacks, so any shop can expose the two hooks Rehearsal needs: create an order, and read an order's state.

Built With

Share this project:

Updates

Submission history