Inspiration

AI shopping agents now pay for people with PayPal, and sometimes they buy the wrong thing. A buyer says "compare prices, don't buy yet", and the agent buys; or it picks the 750 ml bottle when the buyer asked for 500 ml.

Today that buyer opens a dispute. PayPal says it usually decides within 14 days, but some cases take 30 days or longer (PayPal Help), and an item that arrived as described but was not what the buyer asked for may not be covered at all.

PayPal's buyer protection terms already allow a temporary refund during a dispute, recovered after at least 5 business days' notice if the dispute goes against the buyer (Purchase Protection Program). We apply that pattern to AI agent purchases, by rule, and before the check.

Trust the buyer first. If that was wrong, give notice and take it back.

What it does

Refund-First Desk is for companies that run AI shopping agents on PayPal, and for their buyers.

  1. Refund first. The buyer reports the purchase in one sentence. Within the limits, PayPal's refund status is COMPLETED before the purchase is checked.
  2. Then check. A quick check compares the mandate (what the buyer approved) with the PayPal order and closes clear cases. A deep check has Claude read the mandate, the agent's numbered log, the order and the store record, quote its answers, and name the first step where the agent went off the mandate.
  3. Two keys. Fixed rules in code give one verdict, the AI the other. Money moves on its own only when they agree; otherwise a person decides, and nothing is charged meanwhile.
  4. Take it back, only if the buyer was responsible. A notice, 5 business days to dispute, then the saved PayPal wallet is charged again, never for more than was refunded. A dispute stops it. Otherwise the buyer keeps the refund.

A live agent that shops. Claude takes the buyer's instruction, shops a small catalog we wrote one tool call at a time, and pays from the saved sandbox wallet if it decides to. The deep check reads that numbered log. The hasty agent is told it is graded on speed and sees less of the catalog than the careful one; neither is told to make a mistake.

Try it (sandbox, no sign-in; the free Render server can take up to about two minutes to wake): pick Order matches what the buyer approved, press Approve these conditions, Let the agent shop and pay, then Report it. Both keys name the buyer, so a notice comes first; Fast-forward the notice period charges the wallet again. For the live agent: Coffee maker, $45 limit, approve, Run the hasty agent. A live model can end differently, including not paying.

Who it helps, and how risk is bounded

  • Buyers: one sentence, and within the limits PayPal's refund status is COMPLETED in seconds (sandbox).
  • Stores that did nothing wrong: when the agent was at fault, the store is still paid, from the desk's own funds.
  • Agent operators: a case book shows which condition the agent broke, and at which step. In our 30 cases, by our answer key, the agent first left the mandate at its own step in 15 (wrong item 7).

Risk is bounded per buyer. A buyer with no record starts at $50 a case and 2 instant refunds per 30 days. Fixed rules in code move both from the buyer's own cases of the last 180 days: +$25 and +1 refund for each case that was the agent's or the store's fault (at most two, so up to $100 a case and 4 refunds), and -$25 and -1 refund for each case where the buyer was found responsible and charged again, down to $0. The AI never sets the limit. Above it, the check runs first. These numbers are assumptions, set without measured abuse or loss data.

Everything here runs on PayPal APIs that exist today, in the sandbox; on live accounts the re-charge needs PayPal's approval.

How we built it

Node 20 and plain HTML, CSS and JavaScript on the PayPal sandbox:

  • Agent pays, buyer not present: Orders v2 with a Vault vault_id.
  • Refund first: Payments v2 refund with PayPal-Request-Id, then a refund lookup.
  • Re-charge after notice: Orders v2 with the same vault_id.
  • Pay the store: Payouts.
  • Consent withdrawn: Vault payment-token delete.
  • Ledger check: every order is compared with PayPal's capture, refund and payout lookups. The mandate's SHA-256 fingerprint is written into the order's custom_id.

Claude (claude-haiku-5-5) runs both checks and the live agent. Every yes/no answer and first-step answer must quote the record it came from; a quote not found in that record becomes "unknown". Facts that code can compute override the AI, and text inside the records that tries to instruct the checker is treated as data. Tests run against an in-memory PayPal stand-in. The demo is hosted on Render.

Challenges we ran into

  • The two keys are not independent. The rules key computes up to four facts itself and takes the other three from the AI's quoted answers, so a wrong answer backed by a real quote gets past both. We say this in the README.
  • We tuned the prompt on our own 30 cases, so their scores come after tuning. Others wrote new case sets (several writers, and Codex), each answer key fixed before any AI run.
  • One miss repeated. A store added a free sample, and both keys wrongly named the store (the key says the buyer), so the buyer kept the refund and the store went unpaid. A fourth prompt change reads an uncharged free extra as no change to the order, sets one rule for the step that goes over the spending limit, and compares steps one by one. The cases that showed this no longer test it, so we wrote more.
  • Sandbox money. The desk's sandbox balance ran short, so PayPal refused or left refunds pending, and a lookup got a 503 after money had moved. A refund never closes a case or pays a store until PayPal reports it completed; a pending one is looked up again.

Accomplishments that we're proud of

We wrote the 30 prepared cases, their logs and their answer key.

The 30 prepared cases, PayPal sandbox, live Claude (run 7, 2026-11-12):

  • Every refund within the limits completed before the check, in 2.13-4.41 s (26/26).
  • 30/30 verdicts matched the key, none went to a person; first wrong step 29/30.
  • Ledger equals PayPal: 30/30. Buyer charged again when not responsible: 0.
  • Run 8, with the desk's own account added to the ledger check: 30/30 verdicts, refunds in 1.95-3.54 s (26/26), ledger 30/30 for the buyer, the store, PayPal fees and the desk.
  • 9 abuse conditions: no unconsented, duplicate or over-cap charge (9/9).
  • Run 9 (2026-11-13, the fourth change, one run): 30/30 verdicts, first wrong step 30/30, refunds in 2.27-3.29 s (26/26), ledger 30/30 on all four lines.
  • These scores come after tuning on the same 30 cases (untuned, the first wrong step matched 27/30, 25/28 and 27/30). Before the third change, which fixed the order in which the AI checks who is responsible, 2 of 30 verdicts went to a person.

Cases written by others, keys fixed before any AI run:

  • Under the third change, verdicts: 10/10 on 10 (7/10 before it); 39/40 twice on 40 from four separate writers (38/40 before); 12/12 twice on 12 written by Codex (10/12 before; Codex said it read our case file once). The first wrong step matched only 33/40 and 9/12.
  • With the fourth change, on those 52 cases (which showed what to change, so they no longer test it): verdicts 40/40 and 12/12 twice, first step 36 and 37 of 40 and 9 of 12 (six keys follow a different first-step rule). The 30 prepared cases: 30/30 three times. No verdict that was right became wrong.
  • Hold-out 4: 24 new cases, written for this change (the writers were told which shapes to include). They show those shapes are now judged right; they do not show that the prompt generalizes. Verdicts 19 and 20 of 24 before, 23 of 24 twice after; first step 20 and 21 before, 24 and 22 after. One case is wrong in all four runs (key: the buyer; both keys: cannot tell), and its key is open to dispute.
  • Hold-out 5: 20 more cases that were not aimed at any change (the writers got no list of shapes): verdicts 20/20 twice, first wrong step 18/20 twice. It is one set of 20, the writers were Claude too, and the prompt before the fourth change also got 20/20 verdicts on it (first step 16/20).

What trusting first cost: in a 30-case sandbox run where the desk's funds could pay every innocent store (run 8; we wrote the cases and the answer key), the desk's own funds paid $532.74: $468.49 to innocent stores and $64.25 in PayPal fees. $343.20 of that is invoiced to the agent's operator; if it is paid (simulated here), the desk is left with $189.54, or $6.32 a case. We estimate that refunding before the check, instead of after, added $31.93, about $1.06 a case.

Live Claude agent: 12 runs, 8 paid; in all 8 the two keys matched our own reading.

Tests: 194 tests run against an in-memory PayPal stand-in.

What we learned

A refund can come first only because it can be taken back.

Cases written by others, with keys fixed before any AI run, showed what to change next (the free-sample miss).

What's next

On live accounts, charging a saved wallet without the buyer present needs PayPal's approval; that is the first step to production. Then PayPal webhooks and disputes, and limits set from real loss data.

What is real

  • Real PayPal sandbox API calls. Nothing runs on live accounts.
  • The prepared cases replay Claude's recorded answers from run 7, made before the fourth change, and the screen says so. The live agent path and our measurement runs call Claude live.
  • We wrote the 30 prepared cases, the agents' logs, the store records and the answer key, and tuned the prompt on them.
  • Wrong answers remain on cases written by others, and at least one answer key is open to dispute (see the measurements above).
  • Shipping and returns, store records, waiting periods and the agent operator paying its invoice are simulated.

Built With

Share this project:

Updates

Submission history