Dispute Copilot

An AI agent that drafts PayPal dispute responses, with a human approval gate before anything is sent.

Comment

Inspiration

PayPal disputes are a chore for small merchants: read the case, dig up the transaction and tracking, decide whether to fight, write the evidence, and do it before the deadline. An AI agent can do most of that, but sending the wrong thing to PayPal is irreversible, so a human has to stay in charge.

What it does

Dispute Copilot is an inbox of PayPal disputes ranked by urgency and value. For a chosen dispute, an agent reads the case through the Disputes API, pulls the linked transaction and tracking, checks them against the merchant's policy, and drafts a recommendation (contest, accept, or offer a partial refund) with an evidence list. Each evidence claim cites the tool result it came from, and a deterministic 0-100 evidence score (no LLM) shows what is present and what is missing. The merchant reviews the exact PayPal request and approves it. Nothing is sent before that.

The demo video shows a real PayPal sandbox dispute (item not received, $25) drafted by a live NVIDIA model and sent with a real accept-claim call.

How I built it

  • Server-rendered Hono + htmx, TypeScript, SQLite. Runs on 127.0.0.1 by default.
  • The agent only gets a read-only PayPal client. Write methods exist on a separate client that only the approval action calls; a test scans the source to enforce this.
  • Approval gate: CSRF token, bound to draft id and version, atomic claim so a double click cannot send twice, and the approval key is sent as PayPal-Request-Id.
  • Hash-chained audit log of tool calls, drafts, decisions, PayPal requests and responses, with a verify endpoint.
  • Buyer messages reach the model only as delimited untrusted data.
  • Grounding rule: a claim must cite a real tool result, otherwise the score is penalised and the UI warns.

PayPal usage

Customer Disputes API (read, provide-evidence, accept-claim), Payments captures, and Shipping trackers, against the real sandbox. Verified: reads on real disputes, a real provide-evidence write, and a real accept-claim write (PP-R-QEB-10200184 moved to UNDER_REVIEW).

Challenges

  • The buyer-side dispute-create API needs scopes a solo developer cannot get, so test disputes were created by hand in Resolution Center.
  • PayPal rejected an evidence type the dispute had not requested.
  • Making the write path safe against double submits and stale drafts.
  • Keeping the demo honest about what the model actually produced.

Honest limits

  • Offline eval on 13 recorded scenarios (single run, no baseline): the model picked the right action 9 of 13 times (69%), with 2 ungrounded claims. That is why a human approves every action.
  • Not verified against PayPal: file-upload evidence, make-offer, send-message, webhook signature verification. These run in mock mode only (docs/unverified.md lists them).
  • No authentication; single merchant user.

What's next

Read PayPal's requested evidence types and constrain drafts to them, verify the remaining write paths, add carrier lookups, and add authentication and multi-merchant support.

Built With

Share this project:

Updates

Submission history