Rehearsal

Rehearse your PayPal integration for real: describe a business, an AI plans it, and it runs in PayPal's sandbox with every webhook verified. An AI finder catches bugs like duplicate delivery.

Comment

Inspiration

If you build on PayPal, taking the payment is rarely what breaks. The webhooks are: the messages PayPal sends back when an order is paid, refunded, invoiced or disputed. PayPal's Webhook Simulator sends mock events that, in PayPal's own words, cannot be verified through the verify-webhook-signature endpoint, cannot be resent, and do not correspond to any real event or transaction. The dashboard can resend a real event by hand, but you first have to create the transaction and then check your own app yourself. So bugs like fulfilling an order twice when PayPal redelivers a payment event tend to show up later, with real customers and real money. I wanted a way to rehearse a whole business day for real first.

What it does

Rehearsal runs real transactions in PayPal's sandbox and shows each one next to the webhook PayPal sent back for it, checked as genuinely from PayPal.

  • Describe a business, the AI plans it. Type a sentence like "a small clothing brand that sells two shirts by card, refunds part of one order, and invoices a local shop". Claude Sonnet 5 writes a plan of real PayPal calls. Before anything runs, code re-checks the plan against the scenario schema, the spend caps and an allowlist of webhook targets, and you read the exact steps and amounts. The run only starts if it presents that exact plan's fingerprint.
  • Run it live. Card payments, full and partial refunds, invoices (create, send, record payment, cancel), payouts and disputes run in the sandbox. Every webhook is verified through PayPal's verify-webhook-signature endpoint and shown with how long it took to arrive and whether it came out of order.
  • The AI finder catches real bugs. It fills one of four fixed bug-test templates and, for duplicate delivery, asks PayPal to resend a real event. The verdict comes only from the sample store's verified records or PayPal's own read-back, never from the model, and no evidence means INCONCLUSIVE, never OK. It reproduced the classic duplicate-delivery bug (one payment, fulfilled twice) and proved the one-line fix.
  • Real disputes. A dispute needs a real buyer purchase behind it, so Rehearsal keeps a small bank of buyer-approved test purchases, files the dispute through PayPal's sandbox-only dispute API, and checks that PayPal's record names the purchase it used.
  • Replays, CLI and MCP. Anyone can replay recorded runs without a passcode. The same engine runs from a command line and an MCP server, so AI coding assistants like Claude Code can plan and run scenarios against your own app. Nothing writes to PayPal without explicit confirmation.

How I built it

  • Engine: TypeScript on Node 22, a zod scenario schema, an executor that puts an idempotency key on every write and saves resumable run state, and the PayPal sandbox host pinned in code, so it cannot reach live PayPal.
  • AI: Claude Sonnet 5 (claude-sonnet-5) through the Anthropic API, called with the Vercel AI SDK, capped at 4,000 output tokens and 60 seconds a call. The model proposes; code decides.
  • Web console: Next.js 15 and React on Vercel, with Supabase Postgres for run state, the purchase bank and daily quotas. A separate sample store, also on Next.js, receives the webhooks and acts only on events PayPal verifies.
  • PayPal APIs: Orders v2, Payments v2 refunds, Invoicing v2, Payouts v1, Disputes v1, Webhooks v1 (register, verify, list, resend) and Transaction Search v1.
  • MCP server: the MCP TypeScript SDK over stdio.
  • Tests: 1,257 automated tests across the engine, sample store, receiver and console (Vitest, Playwright browser tests, and PGlite to run the SQL against a real Postgres).
  • Built with AI help: Claude Code for implementation and tests and Claude for planning, audits and review, both running Claude Opus 5.5 (Claude Opus 5 for the first two days), plus Claude Design for the console's visual direction. The product itself calls Claude Sonnet 5.

Challenges I ran into

  • Disputes need the buyer's transaction id, which is not in the seller's order or capture records. I found it in Transaction Search, where the buyer payment's reference id matches the seller's capture, and built the purchase bank around that pairing.
  • Timing. Dispute webhooks arrived about 3 minutes after filing in my runs (once 12 hours), and PayPal says new transactions can take up to three hours to appear in Transaction Search. Runs wait 270 seconds when they file a dispute, and anything that does not arrive in time is INCONCLUSIVE, not a pass.
  • Order. In one recorded run, PayPal delivered the second payment's webhook before the first, so the console flags out-of-order arrivals instead of assuming order.
  • A safe public demo. Live runs on the hosted console sit behind a passcode, with spend caps fixed in code, per-session and daily limits, one run at a time, a 10-minute time budget, and webhooks sent only to the sample store.
  • What the sandbox will not do. New disputes start out waiting on the merchant, and the sandbox only allows the require-evidence and adjudicate steps once a dispute is under review, so those stages are out of scope and the site says so.

Accomplishments that I'm proud of

  • The replays and the finder report on the site are real sandbox runs with their real ids: payments, a partial refund and an invoice with every webhook verified, a dispute PayPal reported about 3 minutes later, and webhooks arriving out of order.
  • The finder caught a real class of bug using PayPal's real redelivery and proved the fix, and it never marks anything as passing without the records to back it up.
  • 1,257 automated tests, and the site says plainly what a pass does not prove.

What I learned

  • PayPal retries a webhook up to 25 times over 3 days, which is exactly why a handler must ignore an event it has already seen.
  • Read the docs closely, then test them: several things I assumed about disputes and Transaction Search were wrong until a real run showed me.
  • Let the AI propose and let code decide. The planner is useful because everything it writes is re-checked before it can run.

What's next for Rehearsal

  • Let developers point the hosted console at their own webhook handler, not just the sample store (the CLI and MCP server already can).
  • More finder classes, starting with a dispute against an order that was already refunded.
  • Run scenarios in CI, so every deploy rehearses the business day first.

Runs on

A web app in any modern browser: https://rehearsal-indol.vercel.app. The CLI and MCP server need Node 22 or later.

Built With

  • anthropic
  • claude
  • claude-opus-5.5
  • claude-sonnet-5
  • model-context-protocol
  • next.js
  • node.js
  • paypal
  • paypal-disputes
  • paypal-sandbox
  • paypal-webhooks
  • playwright
  • postgresql
  • react
  • supabase
  • typescript
  • vercel
  • vercel-ai-sdk
  • vitest
  • zod
Share this project:

Updates

Submission history