Inspiration
Merchants lose disputes mostly because evidence is slow to assemble and easy to get wrong. An AI that drafts a response is only useful if the merchant can trust it. A made-up tracking number or a sentence copied from a buyer's message can lose a case or mislead PayPal. I wanted to build the checking layer, not just the drafting layer.
What it does
ProofTrail takes a PayPal sandbox dispute and drafts evidence from the merchant's own order and shipment records. Every claim must cite a record and quote a value, or the draft abstains. Each claim is then labelled VERIFIED, NEEDS_REVIEW or REMOVED. A person reviews the draft and approves the exact payload, bound to its SHA-256 hash. Only then does code submit through the PayPal Disputes API. Submission is a dry run by default, and the code refuses any non-sandbox PayPal URL.
A demo shopping agent also writes a hash-chained, HMAC-signed purchase trail (intent, cart, payment), shown beside disputes about those orders.
How I built it
- Python, FastAPI, SQLite. A poller and a webhook receiver bring disputes in from the sandbox.
- Four verifier layers: structure checks, grounding checked by code (the cited record must contain the quoted value), an independent judge model from a different family, and policy rules (no blame language, no claims resting only on buyer-written text).
- Decision rule in code: a CONTEST needs verified fulfillment evidence and consistent records. Conflicting records force an abstain. For unauthorized-payment disputes it also needs a trusted authorization record.
- Audit log: append-only, enforced by SQLite triggers.
- Models: served through Nebius Token Factory, with a client-side spend cap.
Evaluation
I tested on 52 synthetic cases x 5 repeats, frozen before any test run, across 7 models, with case-level bootstrap 95% intervals. On the main model, decision accuracy was 0.90 for a plain-prompt baseline and 0.99 with the verifier. That gain comes mainly from the conflicting-evidence cases, where a code rule forces abstention, not from the judge model. On weaker models the verifier also cut fabricated claims, such as invented tracking numbers. The verifier has a cost: on the main model it lost a contest it should have made in 2 of 260 runs. All tables, including the unflattering ones, are in the repo.
Challenges
- PayPal's dispute list lagged about 5 minutes behind a new dispute, so the poller is not instant.
- On one sandbox dispute, the buyer page showed a different ID than the seller API returned. The seller API gave 403 for the buyer-page ID, and I don't know why.
- One model sometimes spent its whole token budget on hidden reasoning and returned nothing. I added retries and stopped caching empty replies.
- The scorer over-flagged mentions of attacker text, so I classified flagged runs by hand and published raw and classified counts side by side.
Limits
- All evaluation cases are synthetic and were written by the system's author, so results may not transfer to real disputes. 52 cases is a small sample.
- The webhook receiver is unit-tested but has not been run against a live PayPal delivery. The poller is the working path.
- Accept-claim is not built.
- Two safety rules added after the frozen test run are not covered by those results: claims citing buyer-written text go to review, and unauthorized-payment disputes need trusted authorization evidence.
- The agent trail is tamper-evident, not tamper-proof.
- Sandbox only. Nothing touches live PayPal.
What I learned
A deterministic rule often does more work than a clever judge model, and publishing where the system loses matters more than where it wins.
What's next
Build accept-claim, run the webhook against a live delivery, and test the judge on real disputes.