Payment Truth

Tests a shop's PayPal backend under AI-agent traffic (retries, races, duplicate webhooks). When it breaks, AI finds the cause and the harness proves the fix.

Comment

Inspiration

Agentic checkout is arriving on PayPal: agents that pay on someone's behalf. An agent does not behave like a person at a checkout. It retries when a response is slow, it runs steps in parallel, and it finishes the order while PayPal's own webhook for the same order is still arriving. Most shop backends were written and tested for one person clicking once. I wanted to know what actually breaks when the traffic changes, and to have a fix that is proven, not just suggested.

What it does

Payment Truth drives a shop's PayPal backend with three kinds of agent-shaped traffic:

  • A1 race: the agent completes the order while the CHECKOUT.ORDER.APPROVED webhook arrives at the same time (1, 2 or 4 parallel agent calls, three orderings).
  • A2 retry: the capture succeeds at PayPal but the response is lost, so the agent retries.
  • A3 duplicate webhook: PayPal redelivers the same capture webhook.

After every run it checks four invariants against PayPal's own record and the shop's database:

  • I1 every licence or fulfilment is backed by a COMPLETED capture;
  • I2 at most one grant per order;
  • I3 one shop order per PayPal order;
  • I4 the shop's status agrees with PayPal.

Each verdict is PASS, FAIL or UNKNOWN. Missing data is UNKNOWN, never PASS.

When something fails, an AI model (Claude Opus 5.5) gets the failing verdicts, the event log, PayPal's facts and the handler source, with secrets redacted. It returns findings with line numbers and a corrected file. The harness, not the AI, then decides whether the fix holds: it runs the patched code in a permission-restricted child process (no secrets, no file writes) on every scenario and three seeds, and checks that forged webhooks are still rejected. Only then is the patch marked PROVEN. The result is one screen: before and after verdicts, PayPal's facts next to the shop's records, the AI's findings, the diff, and a downloadable regression test.

Real-world result: evershop

I pointed it at the PayPal integration of evershop, an open-source e-commerce platform (GPL-3.0, about 10.5k stars). Its PayPal code is careful: it re-reads the order before capturing, reuses one PayPal-Request-Id per order, and deduplicates transactions with a unique constraint. Sequential retries and duplicate webhooks passed.

The race failed in all 27 concurrent runs. The return page and the approval webhook both finalized the order, and order_placed (which sends the confirmation email) fired once per caller: 2, 3 or 5 times for one payment. No double charge, but duplicate confirmation emails. The AI pointed at the unlocked existence check on line 68 of finalizePaypalOrder.ts and proposed an 8-line row lock. The harness proved it: 27 of 33 runs failed before, 33 of 33 passed after.

I reported it as evershop issue #1180. Two days later the evershop maintainer merged PR #1193, which "Fixes #1180" with the same row lock plus one more change.

How I built it

  • PayPal: Orders v2 (create, capture, get) with PayPal-Request-Id idempotency, Webhooks (CHECKOUT.ORDER.APPROVED, PAYMENT.CAPTURE.COMPLETED), verify-webhook-signature, and webhook resend. A live sandbox spike checked the behaviour the simulator relies on: the same request id returns the same capture, a different one is rejected with ORDER_ALREADY_CAPTURED, and a resent webhook arrives with the same event id and a valid signature.
  • Scenarios run offline against an in-memory PayPal that follows those observed rules (idempotent capture, signed webhooks, held and released webhook delivery, a network that loses a capture response). This makes every run deterministic and seedable, so a race can be reproduced on purpose.
  • Shops: a small demo shop (SQLite) with a buggy and a fixed handler, and evershop's real PayPal handlers run unmodified against Postgres 15, with only logging, config, plugin hooks and the HTTP client stubbed.
  • AI: Claude Opus 5.5 through OpenRouter for diagnosis and patches. The proof runs under the Node.js permission model.
  • Report and hosting: a self-contained HTML report, rebuilt on every deploy from the recorded runs. The demo-shop patch is re-proven during the build. It runs as a Render static site from render.yaml, so judges need no keys.
  • Plain TypeScript on Node 24 with no runtime dependencies: node:sqlite, node:test and built-in type stripping. 50 tests.

Challenges I ran into

  • Making a race reproducible. Real races are timing luck. I built a held webhook network and seeded orderings so the same race happens every time, and then the fix can be checked against it.
  • Not letting the AI grade its own work. The first diagnosis run had a hint: the planted defects were listed in a comment in the source. I moved them out to docs/planted-defects.md and re-ran hint-free. Proof is decided only by the invariants, in a sandbox with no secrets.
  • Running someone else's code safely and faithfully. For evershop I stubbed as little as possible so the result reflects its real handlers, and only the one file the AI patched is replaced in the "after" run.

Accomplishments that I'm proud of

  • A real bug in a real open-source store, found by the harness, explained by the AI, proven by the invariants, and fixed upstream within two days.
  • On the demo shop, the hint-free diagnosis named 4 causes at the right lines (5 defects were planted), and its single patch still took the shop from 28 of 33 runs failing to 0 of 33, for $0.08.
  • Judges can see it all without keys: a fresh clone runs npm test, check and build:replay offline.

What I learned

Idempotency keys alone do not make a checkout agent-safe. evershop did the PayPal side right; the gap was in the shop's own "first time?" check under concurrency. Payment bugs under agent traffic are often about side effects (emails, licences, fulfilment) rather than double charges, which makes them easy to miss in normal testing.

What's next

  • Run the A1 to A3 scenarios live against the PayPal sandbox, not only the simulator, and verify the approval-webhook signature too.
  • A CI action: run the scenarios on every pull request that touches payment code.
  • More open-source integrations (WooCommerce and others), with each finding reported upstream.

AI tools used

  • Claude Opus 5.5 via OpenRouter: the diagnosis step. It reads failing verdicts, PayPal facts, the event log and the handler source (secrets redacted), and returns findings with line numbers plus a corrected file. The harness proves or rejects that file; the model never decides the verdict. About $0.08 to $0.12 per run.
  • Claude Code: my coding assistant while building the project.
  • Video: narration in an AI clone of my own voice (ElevenLabs); the 12-second paper-collage opening was generated with OpenArt (Seedream 5 Pro, Gemini Omni Flash). Screen footage is real recordings of the live page.

Honesty notes

  • The agent behaviour in the scenarios is my model of how a buying agent acts, not a measurement of any specific agent.
  • The hosted page shows runs recorded on 2026-10-30 and says so; the demo-shop patch is re-proven offline on every build.

Built With

Share this project:

Updates

Submission history