Inspiration

PayPal retries a webhook delivery that is not answered 2xx "up to 25 times over 3 days", so repeats are normal traffic. Its simulator sends mock events that "do not belong to any App", and the resend API re-delivers one real event without saying whether your shop ended up right. Developers find the gap in production:

The first user is a developer who maintains a PayPal checkout plugin or a shop's webhook handler and wants to know, before release, whether repeated, reordered or missing PayPal events leave the shop's books different from PayPal's.

What it does

EchoBench records PayPal webhook deliveries (raw bytes and PAYPAL-* headers), restores the target shop to a saved start point, then replays them duplicated, at the same time, reordered, dropped, late and forged. It judges the shop's books against PayPal's own records, never HTTP codes alone.

When a scenario fails, Claude Sonnet reads the failure, the delivery and the handler, proposes a small patch on a copy, and EchoBench itself runs tsc and re-runs the whole suite on that copy before the patch can be applied. The model never moves money.

A Contract tab turns the rows of the APIMatic Context Plugin's paypal-plan.md (DUPLICATE CLAIMS, UNKNOWN OUTCOMES) into tests. A proxy cuts a refund reply after PayPal already did the refund; the sample shop either retries with the same PayPal-Request-Id (one refund, green) or refunds twice (red).

Honest status of the hosted demo:

  • PayPal runs in sandbox mode. It opens on a recording of real sandbox deliveries (one purchase, two refunds), and all webhook scenarios run on it. The dispute scenario shows "na" there because no real dispute was made.
  • The AI step calls live Claude Sonnet (claude-sonnet-5-5, Anthropic API) with per-IP-hour and per-day caps. If the API fails or a cap is hit, the same action still finishes with a labelled recorded real Claude run or a rule-based stand-in.
  • The demo video was recorded on the hosted app with live Claude.
  • Not run: PayPal resend timing, old original bytes, a forged cert_url postback, simulator events, a real dispute.

How we built it

Built with Claude Code and the APIMatic Context Plugin 0.4.1 (nine TypeScript skills). The APIMatic-generated paypal TypeScript SDK 2.29.0 is the judge's read path (getOrder, getCapturedPayment) and the sample shop's PayPal client; the same scenarios also run through plain REST and must give the same verdicts. The AI step is Anthropic claude-sonnet-5-5 in a ten-tool loop with no PayPal write tool. Stack: Express 5, React 19, Vite, node:sqlite, zod. Evidence is in README.md, VERIFY.md and builtin/sandbox-evidence.json in the repo.

PayPal use, sandbox only: Orders v2 (create, capture, get), Payments v2 (get capture, refund) and Webhooks v1 (signed deliveries, postback verification). AI use: Claude Sonnet diagnoses failures and drafts patches; EchoBench verifies them.

Challenges we ran into

  • captureOrder has an optional body in the SDK, but without one the sandbox answers 415, so we pass body: {}.
  • The SDK adds its own Idempotency-Key, which is not PayPal-Request-Id; only the latter makes PayPal recognise a repeat.
  • Our local stand-in disagreed with the sandbox (repeated request id, overlapping refunds, a dropped custom_id). We fixed it and pinned each case with tests.
  • In the first plugin on/off comparison the model never opened the plugin's skills because the request did not name the SDK. We reran it naming the SDK in both arms.

Accomplishments that we're proud of

  • Real sandbox webhook deliveries recorded on the hosted app and shaken: the judge compares the shop to PayPal's records, not to status codes.
  • The AI patch is never trusted: tsc and a full re-run on a copy must pass before Apply.
  • Five real sandbox contract runs showed the lost-reply refund promise holding with the plan's claim and breaking without it (two real refunds for one request).
  • Plugin on/off, one generation per arm, both with the SDK named: plugin on 10 green, plugin off 9 green of 14 scenarios, and only plugin on passed the two lost-reply checks. It is one sample, not a rate.

What we learned

A 2xx from a handler says little; the books compared with PayPal's own records say a lot. A stand-in is useful only until it disagrees with the sandbox, so we kept measuring both. Verifying a model's patch by running it matters more than reading the model's explanation.

What's next for EchoBench

EverShop on the PayPal sandbox with real deliveries, more plugin on/off generations so the comparison becomes a rate, a resend mode after measuring PayPal's resend timing and signing, and a CI check that fails a build when a handler breaks under replay.

Built With

Share this project:

Updates

Submission history