Inspiration

Around 2008 to 2010 I worked on card disputes at the Royal Bank of Scotland, and I saw how stressful a dispute is for the person on the other side. Since then I've spent more than 15 years in banks such as Credit Suisse, UBS, J.P. Morgan and Société Générale, as an analyst, QA engineer and product manager. For this hackathon I wanted to build something people would actually use, so I started by reading what sellers say on the PayPal Community forum. That's where I found that small sellers are often left alone with disputes.

From the forum I built a persona, Maria, who runs a small ceramics shop: a composite of real sellers, with an AI-generated portrait. The research is in docs/research/target-user-research.pdf.

What it does

Health scan (home page). Three cards: money at risk, sandbox orders missing tracking, and the next reply deadline. Each card opens the rows behind its number. Open disputes are read live from the PayPal sandbox, and the page shows an error instead of a zero if PayPal can't be reached.

Tracking fixer (/orders, AG Grid). A grid of orders with filter chips, a quick filter, status pills, totals for the rows shown. The seller checks what will be sent, adds a carrier and tracking number, and confirms. The app calls PayPal's Shipment Tracking API, and a row shows "Tracking added" only when PayPal's reply lists that order. Seeded rows are labelled "Demo data" and can't be sent.

Dispute cases (/cases). Each case gets a strength: Strong, Weak or Mixed (the seller decides). Rules in code decide it from PayPal's data, the order's tracking and the seller's own facts on the case card, such as the delivery date on the carrier page or whether the item matches the listing. Where a rule applies, the card shows the rule's own reasons. Where the rules leave the case open, Claude decides and writes the reasons, and each reason must cite the case facts it relies on. The card shows the steps for the case, the conversation with the buyer and a reply draft. The seller edits and approves, and the app sends it through PayPal's Disputes API: a message to the buyer, the order's tracking as evidence, or a refund. Each button is on only in the PayPal stage that accepts it.

How we built it

I'm a product manager, and I built this with Claude as my engineering team. I wrote the PRD, made every product and policy decision, and reviewed and tested the result. Claude wrote most of the code under my direction.

The part I spent most time on is the dispute rules. They come from PayPal's Seller Protection terms and from how disputes go in real life. I decided each rule myself, for example:

  • No tracking on an item-not-received case means refund.
  • "The item matches the listing" counts only with evidence, because it's the seller's view and PayPal decides on evidence from both sides.
  • A missed reply deadline means Weak, because protection needs a reply in time.

Each rule is in the code with my name and the date I decided it. I also signed off the expected answer for every test case in the golden set. When a case looked wrong to me, I changed the rule rather than the test.

I had the product reviewed four times for gaps in logic and coherence, and decided how to fix each one. One review showed that the code rules decide most test cases before the AI's answer counts, so I now report the AI and the rules separately.

Built with Next.js and TypeScript, the PayPal Disputes, Shipment Tracking and Transaction Search APIs (sandbox), AG Grid Community for the orders grid, Claude Opus 5.5 for the case reasons and Claude Sonnet 5.5 as the judge. I used Postman to try the PayPal sandbox calls by hand before building them into the app.

Challenges we ran into

  • Creating disputes in the sandbox. The API to open disputes needs a permission the sandbox app doesn't have. So I opened the six sandbox disputes by hand from the sandbox buyer account, and the app reads and answers them through the Disputes API. I didn't want a single faked API call.
  • Each PayPal action works only at certain stages. A message to the buyer, evidence and a refund are each accepted only in some stages of a case. The app reads the case status first, and turns each button on only when PayPal accepts it.
  • Keeping the AI to the facts. In early runs the AI added evidence examples that weren't in the case, and once it gave a fact to the wrong person. Every reason now has to name the case facts it relies on, a guard in code drops anything else, and a second model checks each reason.
  • My own rules. Three of my first rules were wrong: a missed deadline was "Mixed" instead of Weak; "the item matches the listing" counted as proof, although it's only the seller's view; and the reply buttons switched off when PayPal sent no deadline, which blocked sellers who could still reply. I found them while signing off the test cases, and fixed the rules, not the tests.

Accomplishments that we're proud of

  • A real end-to-end action: on sandbox case PP-R-JEX-10197437, the app sent the order's tracking as evidence through the Disputes API, PayPal accepted it, and the case moved to waiting for the buyer.
  • 21 out of 21 test cases right, in 4 out of 4 runs, with every expected answer signed off by me.
  • A checker I can trust: it agrees with my own labels on 58 of 58 reasons, and caught all 12 false reasons I planted.
  • Nothing goes to PayPal without the seller's click, and the app never shows a success that PayPal didn't confirm.
  • As a product manager, I built a working product in about a week, with Claude as my engineering team.

What we learned

  • Check the checker. An AI judge is only useful if you test it with mistakes you know are there.
  • Measure honestly. I was proud of 19/19, until a review showed my own rules decided most of those cases before the AI's answer counted. Now I report the AI and the rules separately, so the number means what it says.
  • Rules where policy is clear, AI where it isn't. Seller Protection is decided in code, so the most important answer is predictable and explainable. The AI weighs only the cases the rules leave open.
  • The seller's word isn't evidence. PayPal decides on evidence from both sides, and the app has to treat it the same way.

What's next for Seller Shield

  • A pilot with 15 real sellers. Including a check of each seller's disputes from the last 90 days: would the recommendation have been right?
  • Live PayPal and real shops. Moving from the sandbox to live accounts, with PayPal partner onboarding and real Shopify and WooCommerce connections.
  • Deadline alerts by email before a reply is due.
  • "Refund after return" as an option for not-as-described claims.
  • Pricing: a flat €15–19 a month rather than a cut of recovered money, to be tested in the pilot.

How I check the AI

  • Golden set: 21 cases with answers I signed off. 17 decided by my rules, 4 by the AI, all right in 4 out of 4 runs.
  • Judge (Claude Sonnet 5.5): every reason supported by the case facts in 21 of 21 cases; agrees with my own labels on 58 of 58 reasons; caught 12 of 12 false reasons I planted.
  • Grounding guard in code: 0 reasons dropped on the sandbox and demo disputes. 233 unit tests pass.
  • Full scorecard: docs/eval-scorecard.md

What is real and what is demo

  • Real: six PayPal sandbox disputes (four closed, two open), five real sandbox orders, and the tracking evidence PayPal accepted on PP-R-JEX-10197437.
  • Demo: 200 seeded orders and one demo case (PP-R-DEMO-25), labelled as demo on screen, with sending off.

One example, end to end

Facts the AI was given (no names or addresses): reason "item not received", a tracking number added, not marked as delivered, status "waiting for the buyer", the seller cannot reply right now, amount EUR 65.

Reasons it wrote (each one names the facts it used):

  • The buyer says the item was not received. (reason)
  • A tracking number was added, but the item is not marked as delivered. (hasTracking, delivered)
  • PayPal's Item Not Received rule needs a delivered status, so this case does not meet it. (reason, delivered)
  • The dispute is waiting for the buyer's response, and you cannot respond right now. (status, canRespond)
  • The disputed amount is EUR 65. (amount)

The app rated it "Your decision" with low confidence, because the rules don't settle it. The judge (Claude Sonnet 5.5) checked each reason against these facts; the same check passes on all 21 golden cases.

Limits

  • Sandbox only, no live money or buyers. Photos and documents still go through the PayPal Resolution Center.
  • The AI sees only case facts, never buyer names, emails, addresses or messages. Reply drafts are templates, not AI-written.
  • The judge is also an AI model, so its scores support the other checks rather than prove them.

Built With

  • ag-grid
  • anthropic
  • claude-opus-5.5
  • claude-sonnet-5.5
  • gitlab
  • next.js
  • node.js
  • paypal
  • paypal-disputes-api
  • paypal-sandbox
  • paypal-shipment-tracking-api
  • paypal-transaction-search-api
  • react
  • render
  • typescript
Share this project:

Updates

Submission history