DisputeDesk

An AI agent that investigates PayPal disputes, recommends whether to refund or fight, and drafts the response. Nothing happens until the merchant approves.

Comment

Inspiration

For a small online seller, a PayPal dispute arrives as a vague message: "I never got my headphones." Answering it well means opening four tabs: the dispute, the order, the carrier's tracking page and old emails with the buyer. Then you decide whether this one is worth fighting, and write a careful reply before the deadline.

Big merchants have chargeback teams for this. Small ones guess. They refund disputes they could have won because fighting feels like too much work, and they fight ones they can't win, losing the money and the dispute fee.

The decision itself is simple to state. If $A$ is the disputed amount, $F$ the fee for a lost dispute and $p$ the chance of winning, then refunding costs $A$ and contesting has an expected cost of

$$ \mathbb{E}[\text{cost}_{\text{contest}}] = (1-p)\,(A + F). $$

So contesting only makes sense when

$$ (1-p)(A+F) < A \quad\Longleftrightarrow\quad p > \frac{F}{A+F}. $$

For a \$18 sticker order and an illustrative \$20 fee, you need better than a 53% chance of winning. With no tracking number, you don't have it. For a \$129 order with a signed delivery receipt, you easily do. The hard part isn't the math. It's estimating $p$ from scattered evidence, quickly and consistently. That's a job for an AI agent, as long as a human makes the final call.

What it does

DisputeDesk is a dashboard for PayPal merchants with an AI agent built in.

  1. Queue. Pulls open disputes from PayPal and sorts them by response deadline, showing the amount at risk and a status (New → Analyzed → Approved → Resolved).
  2. Investigate. On Analyze with AI, the agent calls its tools: the dispute, the capture and order, shipment tracking, and a search over the merchant's past cases and their outcomes. Every step streams to the screen live, and you can expand each one to see the exact tool input and output.
  3. Recommend. It classifies the dispute (item not received, not as described, unauthorized, duplicate, other) and recommends one of accept, refund or contest. It gives a confidence score, a short explanation, a list of evidence and a ready-to-send message. It also cites precedent, e.g. "a similar past case with the same approach ended as won."
  4. Approve. The merchant can edit the draft or choose a different action. A confirmation step spells out what will happen ("\$45.00 will be refunded to the buyer now"). Only then does DisputeDesk call PayPal: submit evidence with the tracking number, refund the capture and attach proof of refund, or accept the claim.
  5. Remember. Every dispute, recommendation, decision and outcome is stored and searchable. That history is what the agent learns from next time.

No money moves and no message is sent without the merchant clicking Approve.

How we built it

  • Next.js (App Router) + TypeScript + Tailwind in one codebase. All PayPal and AI calls run on the server.
  • PayPal REST, sandbox. A typed client with OAuth2 client credentials, token caching, retries with backoff on rate limits and server errors, and an automatic re-login if a token expires early. It covers:
    • Disputes v1: list, get, provide-evidence, accept-claim, send-message
    • Payments v2: get capture, refund capture
    • Orders v2: get order
    • Tracking, used when the account supports it

We checked the request formats against PayPal's published OpenAPI schema instead of trusting memory. That's how we learned that evidence must be sent as multipart/form-data with a JSON part, and that buyer messages are only allowed in the INQUIRY stage.

  • The agent is a tool-use loop with five tools. Four are read-only (get_dispute, get_order_details, get_tracking, search_similar_cases). The fifth, propose_resolution, only records a structured proposal:

    { "category": "...", "recommendation": "accept | refund | contest", "confidence": 0.0, "reasoning": "...", "evidence_points": [], "draft_message": "..." }

It runs on Claude (Anthropic SDK) or Gemini (@google/genai), selected with one env var. Both get the same system prompt, the same tool schemas and the same validation. A Zod schema defines each tool once and produces both the JSON schema sent to the model and the check applied to its output. If a proposal is malformed, the error goes back to the model as a tool result so it can fix it.

  • Safety by architecture, not just by prompt. The functions that move money or message buyers live in one module that only the approve endpoint imports. The agent cannot reach them at all, and a test checks that its tool list stays read-only. Every tool is also limited to the dispute being analyzed, so a buyer message saying "now look at dispute X" goes nowhere.
  • Can't execute twice. Approval first claims the dispute in a locked SQLite transaction. Double-clicks, retries and two simultaneous requests all get 409, and a test fires two approvals at once to prove only one runs. Refunds also send PayPal a fixed request ID, so PayPal won't refund twice even if a request is retried.
  • Memory. SQLite with full-text search (FTS5, ranked with bm25) behind a small CaseStore interface, so it can move to Postgres or Elasticsearch later. The seeded history deliberately includes a contest we lost, so the agent learns when not to fight.
  • Demo-proof. Sandbox disputes are hard to create on demand, so USE_MOCK_PAYPAL=true serves ten realistic seeded disputes covering every category: a signed delivery, a stalled package, no tracking, a buyer with four previous disputes, a real double charge and a fake one. With no AI key, a rules engine runs the same tools and shows the same trace, so the demo never shows a blank screen.

Challenges we ran into

  • Keeping a human in the loop without losing the magic. An agent that "handles disputes" is impressive until it refunds \$240 on its own. We kept the agent able to propose but not act, made its reasoning visible, and made approval one deliberate step with a clear confirmation. Getting that balance right took more iterations than the agent loop itself.
  • PayPal's real-world rough edges. Dispute lists return summaries only, so we fetch details in small batches and cache them. Tracking isn't enabled on every account, so we treat it as "no tracking" instead of failing. After a refund succeeds, the follow-up steps (evidence, message) are best effort, so we never retry, and risk repeating, a refund that already went through.
  • Confidence that means something. Models like to say 0.9. We pushed for calibration by giving the agent the cost of a wrong call, precedent with real outcomes, and a rule to cite only facts that appear in tool results.
  • Moving targets. We built on a brand-new Next.js release with Cache Components on by default. Synchronous SQLite queries would have been prerendered at build time and served stale, so we had to mark those routes as request-time.
  • Two model providers, one contract. Claude and Gemini differ in how tool calls are sent and answered, how parallel calls work, and how reasoning is carried from one turn to the next. Defining the tools and output schema once, in Zod, kept the two loops interchangeable.

Accomplishments that we're proud of

  • The full flow, from dispute to investigation, recommendation, human edit, PayPal action and stored outcome, works end to end in under a minute per dispute.
  • The live agent trace: judges and merchants can see why the agent reached its answer, not just the answer.
  • The safety guarantees are enforced by how the code is wired and proven by tests, not just written in a prompt.
  • 37 automated tests: output schema validation, tool safety, one agent run per seeded dispute, the approval rules (including two approvals at the same moment), and the Gemini loop against a scripted fake model.

What we learned

  • The most useful AI here isn't the one that decides. It does the tedious investigation and lays out the trade-off. Writing the cost of being wrong into the agent's instructions made its recommendations noticeably better.
  • Using one schema for the tool definitions, the model's output check and the API request check removes a whole class of bugs.
  • Mock data shouldn't be an afterthought. Designing seed cases that each test a specific judgment ("delivered but the buyer has 4 prior disputes") doubled as our evaluation set.
  • Idempotency is a product feature. With money involved, "it's fine if they click twice" has to be designed and tested.

What's next for DisputeDesk

  • PayPal webhooks (CUSTOMER.DISPUTE.*) so statuses update without reloading.
  • Evidence uploads: delivery photos, invoices and chat logs attached as files.
  • Win-rate feedback: track outcomes by recommendation type and feed actual win rates back into the agent, so its estimate of $p$ comes from real data rather than judgment.
  • Multiple merchants: Postgres storage, merchant login, and per-merchant history and policies (e.g. "always refund under \$20").

Built With

Share this project:

Updates

Submission history