Inspiration
A dispute response is a place where a wrong detail can weaken a merchant's case, and language models sometimes state wrong details with confidence. I wanted to see how far I could get with a simple rule: the AI may draft, but nothing reaches PayPal unless plain code has checked it against data PayPal already holds.
What it does
Dispute Defender helps a merchant answer a PayPal "item not received" dispute.
- The app reads the dispute from the PayPal sandbox.
- Claude (on Amazon Bedrock) drafts the response as structured claims: transaction ID, amount, date, seller protection, tracking number and carrier, each with a value and a source.
- Deterministic code verifies every claim. Four are checked against PayPal's dispute data; tracking and carrier are checked against the merchant's own record. Each claim comes out VERIFIED, FLAGGED or BLOCKED.
- Only VERIFIED claims are written into the response text, using a fixed template. The merchant reviews it on a local web page and approves.
- The evidence is sent to the PayPal sandbox, and the app reads the dispute's new state back from PayPal instead of assuming it.
A replay mode lets anyone try the full flow with no PayPal or AWS credentials, on clearly labelled simulated disputes (including one where a wrong amount is injected on purpose and blocked).
How we built it
- Python, FastAPI and server-rendered HTML for the approval page (no JavaScript, no database).
- A direct REST client for the PayPal Disputes API in the sandbox: list, show and provide-evidence.
- Claude Sonnet 4.6 on Amazon Bedrock through the Converse API with a forced tool call, so the model can only return structured claims. Output is validated with pydantic, and buyer messages are treated as untrusted text.
- The verifier, the response renderer and the send gate are plain code. The gate refuses the send unless every condition holds, and the server re-reads and re-verifies at approval time. The page uses single-use CSRF tokens and binds to localhost only. Dry run is the default.
- A small evaluation on 24 simulated scenarios: 144 live Bedrock calls (two models, three repetitions each), with scenarios, scoring and prompts frozen by hash before the first call, plus 1,192 offline fault-injection cases.
- 714 offline tests.
- I used Claude Code as a coding assistant and reviewed each plan and change before merging.
Challenges we ran into
- The sandbox could not settle a dispute: the adjudicate call was refused every time, so the project ends at "PayPal accepted the evidence", not at "won".
- Dispute states change on their own. After the evidence, the state went from UNDER_REVIEW to WAITING_FOR_BUYER_RESPONSE minutes later, so the app always reads the state back.
- PayPal's
allowed_response_optionsdid not match the actions it offered, so the send gate relies on the provide-evidence link. - The dispute data has no shipping fields, so tracking and carrier can only be checked against the merchant's own record. That creates a real blind spot.
- Evidence content is not echoed back, so the read-back confirms the new state, not what PayPal stored.
Accomplishments that we're proud of
- A real end-to-end run against the PayPal sandbox: draft, verify, approve, send, read back.
- On the simulated scenarios, neither model made a claim error in 390 claims, and the verifier caught all 1,065 faults injected after the fact (expected, since they were written knowing its rules).
- Reporting the weak spot plainly: when the merchant's own record is wrong, the verifier does not catch it (0 of 43 simulated cases).
- A path for anyone to try it with no credentials, and a README that states every limit.
What we learned
- Verification works best outside the model, with a rule the model cannot argue with.
- "Unsupported" is different from "wrong": one simulated scenario produced 6 claims per model that would have gone out without the verifier, and none of them were model errors.
- With identical outputs across repetitions, the effective sample is 24 cases per model, not 72, and zero observed errors is not a zero error rate.
What's next for Dispute Defender
- Close the blind spot by checking tracking against the carrier's own data instead of only the merchant's record.
- Support more dispute reasons and carriers (today: item not received, FedEx).
- Retry after an expired access token, and test more dispute stages.
- Evaluate on more disputes than the 24 simulated scenarios. Everything so far is sandbox-only.
Built With
- amazon-bedrock
- amazon-web-services
- anthropic
- boto3
- claude
- claude-sonnet
- fastapi
- human-in-the-loop
- jinja
- llm
- paypal
- paypal-disputes-api
- paypal-sandbox
- pydantic
- pytest
- python
- rest-api
- structured-output
- uvicorn
