Tamper

AI agents audit your own PayPal sandbox checkout for money-losing mistakes, prove each one with PayPal's own records, and re-run the check to prove your approved fix holds.

Comment

Inspiration

The PayPal integration mistakes that cost money don't throw errors. A shop that ships on the amount the browser reported, marks an order paid because the return page loaded, credits any POST to its webhook URL, or counts a redelivered event twice passes every happy-path test. Then it loses money in production. PayPal's docs warn about each of these: verify on the server, verify webhook signatures, be idempotent. But nothing tells you whether your checkout makes them. We wanted a test that answers with PayPal's own records, not an opinion.

What it does

Tamper audits a PayPal checkout running in the sandbox. A team of cooperating agents works through it, and you stay in charge of every change:

  1. Recon (Claude) reads the shop's source and maps where it decides an order is paid.
  2. Auditors drive real sandbox orders through the checkout the way a careless client would: a changed amount, a "paid" signal straight from the browser, a webhook PayPal never sent, and PayPal's own genuine webhook delivered a second time. Then an AI shopping assistant uses the shop's agent tools: it passes along a price it read elsewhere, and it retries checkout.
  3. The Judge (no AI, deterministic) confirms a finding only if PayPal's own record shows it, using the capture, the order and verify-webhook-signature. Every finding carries the sandbox order, capture and webhook IDs that prove it.
  4. The Patcher (Claude) names the one safety change that fixes each finding.
  5. You approve or reject each fix.
  6. The Verifier rebuilds the shop with only that change and re-runs the same check against PayPal. A wrong fix fails verification, and there is a test for that.

The dashboard streams each agent's steps live and ends with a scorecard. On the demo shop, all 6 checks find a real issue and all 6 fixes verify.

The six checks

  • Amount mismatch: ships on a price the browser reported, not what PayPal captured. Proof: the capture.
  • Browser-trusted "paid": marks an order paid without asking PayPal. Proof: the order status.
  • Unverified webhook: credits a "payment completed" event without checking it came from PayPal. Proof: PayPal's signature check returns FAILURE.
  • Webhook replay: credits the same capture twice when an event arrives again. Proof: the credited amount against the captured amount.
  • Agent price: the shop's agent tool charges a price the AI assistant sends instead of the catalog price. Proof: PayPal captured $1.00 for $49.00 of goods that shipped.
  • Agent retry: an assistant retries checkout and the shop opens a second PayPal order for the same cart. Proof: two completed PayPal orders for one cart.

The replay check uses PayPal's real event. On the hosted demo it waits for PayPal's own signed PAYMENT.CAPTURE.COMPLETED delivery, has the Judge confirm with PayPal that it's genuine, and then delivers that exact event again. A genuine event is still genuine the second time, so checking signatures doesn't prevent the double credit. Only de-duplicating on the capture ID does.

Tamper moves no money of its own. It reads PayPal's records and drives sandbox orders on a demo shop we own.

How we built it

  • PayPal REST APIs, sandbox only, on a dedicated app:
    • Orders v2: create, capture, and show order.
    • Payments v2: show capture.
    • Webhooks: verify-webhook-signature and webhook lookup.
    • A webhook endpoint that receives PayPal's real deliveries.
  • Claude (Sonnet 5.5, Anthropic SDK) powers Recon and the Patcher. The Patcher must answer in JSON naming one of three safety switches. The Verifier flips exactly that switch, so the model's answer is tested, not trusted.
  • The demo shop is one implementation with five switches: verify with PayPal before shipping, verify webhook signatures, de-duplicate captures, take prices from the catalog, and run one checkout per cart. With all five off, it is the weak shop.
  • Agent tools shaped like WebMCP tools (search_products, add_to_cart, start_checkout, fulfill_cart, each with a name, description, input schema and execute). The Auditors call them the way an AI shopping assistant would. The retry fix sends the cart id as PayPal-Request-Id, PayPal's own idempotency key.
  • Stack: Node 22 with no framework, and server-sent events for the live agent log. Approvals are bound to the run that asked for them. Hosted on Render.
  • Tests: 54 unit tests with PayPal and the model faked. Every bug fix comes with a test that failed first.

Challenges we ran into

  • Verifying PayPal's own webhook. verify-webhook-signature checks the event exactly as PayPal sent it, so Tamper keeps the raw body and splices it into the request byte for byte rather than re-serializing it. PayPal's real deliveries now verify SUCCESS on the hosted demo.
  • Making the AI accountable. A fix described in prose proves nothing. The Patcher has to name a concrete switch, and the Verifier re-runs the check with only that switch on.
  • Never claiming more than happened. A local run with no PayPal webhook replays a synthetic stand-in, and the proof says so ("synthetic stand-in, not PayPal-signed"). A re-check that shows nothing either way is reported as pending, never as "fix holds".
  • PayPal's idempotency key isn't enough on its own. A repeat with the same PayPal-Request-Id gets PayPal's same order back, but in the sandbox, two same-key calls at the same moment created two orders in 3 of 5 tries. So the retry fix also runs one checkout per cart at a time, and the test double models that race.
  • A public demo that creates real sandbox orders.
    • One audit runs at a time, and each visitor gets a few runs an hour.
    • The visitor is counted by Cloudflare's connecting IP. We found that on our host, the first X-Forwarded-For entry is whatever the client sends.
    • Only the dashboard that started a run can approve its fixes.
    • A fix nobody decides on within 5 minutes is rejected, so an idle tab can't hold the slot.

Accomplishments that we're proud of

Every number on the scorecard traces back to a PayPal sandbox record. The Judge contains no AI at all. PayPal's genuine signed webhook, verified with PayPal, is part of the live demo. A wrong fix from the model can't pass, because the Verifier re-runs the check with only that fix applied.

What we learned

Verifying signatures and de-duplicating events are different defenses, and a shop needs both. AI is most useful here reading code and proposing a fix. Deterministic code and PayPal's records make the call.

What's next

  • Audit any developer's sandbox checkout, not only our demo shop, using their own sandbox app.
  • Register the agent tools with the browser's WebMCP API once it settles, and add checks for product text that steers an assistant.
  • More checks: currency and payee mismatch, double capture under concurrency, and refund and dispute webhooks.
  • A CI mode that fails the build when a check confirms.
  • Opening the approved patch as a pull request.

Built With

Share this project:

Updates

Submission history