Inspiration

A dispute response is a place where a wrong detail can weaken a merchant's case, and language models sometimes state wrong details with confidence. I wanted to see how far I could get with a simple rule: the AI may draft, but nothing reaches PayPal unless plain code has checked it against data PayPal already holds.

What it does

Dispute Defender helps a merchant answer a PayPal "item not received" dispute.

  1. The app reads the dispute from the PayPal sandbox.
  2. Claude (on Amazon Bedrock) drafts the response as structured claims: transaction ID, amount, date, seller protection, tracking number and carrier, each with a value and a source.
  3. Deterministic code verifies every claim. Four are checked against PayPal's dispute data; tracking and carrier are checked against the merchant's own record. Each claim comes out VERIFIED, FLAGGED or BLOCKED.
  4. Only VERIFIED claims are written into the response text, using a fixed template. The merchant reviews it on a local web page and approves.
  5. The evidence is sent to the PayPal sandbox, and the app reads the dispute's new state back from PayPal instead of assuming it.

A replay mode lets anyone try the full flow with no PayPal or AWS credentials, on clearly labelled simulated disputes (including one where a wrong amount is injected on purpose and blocked).

How we built it

  • Python, FastAPI and server-rendered HTML for the approval page (no JavaScript, no database).
  • A direct REST client for the PayPal Disputes API in the sandbox: list, show and provide-evidence.
  • Claude Sonnet 4.6 on Amazon Bedrock through the Converse API with a forced tool call, so the model can only return structured claims. Output is validated with pydantic, and buyer messages are treated as untrusted text.
  • The verifier, the response renderer and the send gate are plain code. The gate refuses the send unless every condition holds, and the server re-reads and re-verifies at approval time. The page uses single-use CSRF tokens and binds to localhost only. Dry run is the default.
  • A small evaluation on 24 simulated scenarios: 144 live Bedrock calls (two models, three repetitions each), with scenarios, scoring and prompts frozen by hash before the first call, plus 1,192 offline fault-injection cases.
  • 714 offline tests.
  • I used Claude Code as a coding assistant and reviewed each plan and change before merging.

Challenges we ran into

  • The sandbox could not settle a dispute: the adjudicate call was refused every time, so the project ends at "PayPal accepted the evidence", not at "won".
  • Dispute states change on their own. After the evidence, the state went from UNDER_REVIEW to WAITING_FOR_BUYER_RESPONSE minutes later, so the app always reads the state back.
  • PayPal's allowed_response_options did not match the actions it offered, so the send gate relies on the provide-evidence link.
  • The dispute data has no shipping fields, so tracking and carrier can only be checked against the merchant's own record. That creates a real blind spot.
  • Evidence content is not echoed back, so the read-back confirms the new state, not what PayPal stored.

Accomplishments that we're proud of

  • A real end-to-end run against the PayPal sandbox: draft, verify, approve, send, read back.
  • On the simulated scenarios, neither model made a claim error in 390 claims, and the verifier caught all 1,065 faults injected after the fact (expected, since they were written knowing its rules).
  • Reporting the weak spot plainly: when the merchant's own record is wrong, the verifier does not catch it (0 of 43 simulated cases).
  • A path for anyone to try it with no credentials, and a README that states every limit.

What we learned

  • Verification works best outside the model, with a rule the model cannot argue with.
  • "Unsupported" is different from "wrong": one simulated scenario produced 6 claims per model that would have gone out without the verifier, and none of them were model errors.
  • With identical outputs across repetitions, the effective sample is 24 cases per model, not 72, and zero observed errors is not a zero error rate.

What's next for Dispute Defender

  • Close the blind spot by checking tracking against the carrier's own data instead of only the merchant's record.
  • Support more dispute reasons and carriers (today: item not received, FedEx).
  • Retry after an expired access token, and test more dispute stages.
  • Evaluate on more disputes than the 24 simulated scenarios. Everything so far is sandbox-only.

Built With

  • amazon-bedrock
  • amazon-web-services
  • anthropic
  • boto3
  • claude
  • claude-sonnet
  • fastapi
  • human-in-the-loop
  • jinja
  • llm
  • paypal
  • paypal-disputes-api
  • paypal-sandbox
  • pydantic
  • pytest
  • python
  • rest-api
  • structured-output
  • uvicorn
Share this project:

Updates

Submission history