DisputeDefender

An AI co-pilot that helps PayPal merchants respond to disputes with evidence that is always grounded in real data.

Comment

DisputeDefender: Every Claim Deserves Evidence

An evidence-grounded AI copilot that helps PayPal merchants decide which disputes to fight, draft responses that cite real records, and approve every action themselves.

Inspiration

A customer opens a dispute, and the clock starts. The evidence is scattered across orders, tracking pages, shipping policies, and customer messages. A large company hands this to a dedicated team. A small merchant has to search through all of it while still running the business.

The obvious fix is an AI that writes the response. We saw the problem with that: a language model can confidently invent a delivery confirmation or a tracking number that never existed. In a payment dispute, that does not just give a wrong answer. It can sink the merchant's case.

So we asked a different question: what would an AI copilot look like if evidence came first? Facts come from the merchant's own records, every claim points back to one of them, the financial decision is explainable, and a human approves anything that moves money.

What it does

DisputeDefender turns a PayPal dispute into a structured workflow:

  1. Detect. Verified PayPal webhooks and the Disputes API bring in each case, and a dashboard shows open cases by deadline.
  2. Reconstruct. The app builds a fact base from orders, payments, shipment tracking, store policies, and customer messages. Each fact gets a stable ID.
  3. Decide. A deterministic engine scores the evidence and compares the estimated cost of fighting with the cost of refunding. It recommends fight, refund, or accept, with the formula visible. The LLM does not estimate win probabilities.
  4. Draft. When fighting makes sense, Google Gemini writes the response from the facts. Every paragraph cites fact IDs. A validator checks amounts, dates, tracking numbers, and references. If the draft still fails validation after a retry, the app falls back to a deterministic draft.
  5. Approve. The merchant reviews the exact text and approves it. Any edit reruns the grounding check and invalidates the earlier approval, which is tied to a hash of the draft. Evidence submission, claim acceptance, and refunds all require explicit approval.
  6. Record. Agent decisions, merchant actions, PayPal calls, and state changes go into an audit log. A read-only case assistant, built on the PayPal Agent Toolkit, answers questions about a case without changing anything.

The app also generates a PDF evidence packet from the cited facts and checks deadlines on a schedule.

How we built it

The main design choice was splitting responsibilities. The AI handles language. Deterministic code handles money and validation. People authorize actions.

  • App: Next.js, React, and TypeScript. AG Grid powers the case queue, and AG Charts shows operational metrics.
  • Data: PostgreSQL with Drizzle ORM, storing orders, disputes, facts, drafts, approvals, and the audit log.
  • PayPal: Disputes API (retrieve cases, submit responses), Orders and Payments (refunds), Shipment Tracking (delivery evidence), Webhooks (signature verification and event deduplication), and the Agent Toolkit (read-only assistant tools). Write requests send a PayPal-Request-Id for idempotency.
  • AI: Gemini through the Vercel AI SDK, with structured output validated by Zod. The drafting and analysis calls have no tools.
  • Evidence PDF: pdf-lib.
  • Testing: Vitest for unit and integration tests, and a Playwright end-to-end test for the dispute flow. Sandbox scenarios cover strong evidence, weak evidence, different dispute reasons, and a prompt-injection attempt.

The pipeline:

PayPal webhook → verified intake → fact extraction → deterministic analysis → AI draft → grounding validation → merchant approval → PayPal action

Our most important decision was not to treat the model as the final authority. The validator checks generated claims against stored records. Buyer messages are treated as untrusted input: they can be shown as context, but they never become citable evidence.

Challenges we ran into

Accuracy over fluency. Language models are good at convincing prose, but disputes depend on exact dates, amounts, and identifiers. We separated drafting from validation. The model proposes structured claims with references, and code checks them. An unsupported claim cannot pass just because it sounds reasonable.

Knowing when not to fight. The most persuasive response is not always the best business decision. Instead of asking an LLM for a win probability, we built an explicit evidence score and cost comparison. The strength bands are heuristics, not validated probabilities, and the UI says so.

Hostile input. Buyer messages are useful context, but they are also text an attacker controls. We isolated them as data and built a demo case that tries to manipulate the assistant.

Automation without losing authority. Generated drafts could easily turn into executed decisions. Approval gates, draft-specific approval hashes, and re-validation after every edit prevent that.

Imperfect external systems. Duplicate webhooks, API failures, missing evidence, model outages, and deadlines all had to be handled. Reproducible sandbox cases and a deterministic fallback let us test these paths without depending on live model output.

Accomplishments that we're proud of

  • A grounding validator that rejects the AI's own output. Inserting a fake tracking number into a draft blocks approval.
  • A clear split between probabilistic and deterministic parts. The model writes language. Code does the scoring, the cost math, the state transitions, and the validation. That makes the recommendation testable and open to challenge.
  • Human approval that means something. The merchant approves the exact text, and edits invalidate that approval. Nothing is submitted or refunded silently.
  • Demo cases that show bad outcomes too, including weak evidence, unfavorable disputes, and prompt injection.
  • Graceful failure. When generation is unavailable or fails validation, the workflow falls back to a deterministic path instead of inventing content.

What we learned

Reliable AI in a sensitive workflow depends more on boundaries than on better prompts. Structured outputs, source references, deterministic validators, approval gates, and audit trails are core features, not extras.

Explainability does not require showing model reasoning. It can mean showing the evidence used, the assumptions behind a recommendation, and the exact action the user is approving.

Working with PayPal's sandbox also reinforced how much idempotency, webhook security, and consistent state matter when external systems are imperfect.

What's next

DisputeDefender is currently a sandbox prototype. Next steps:

  • More dispute reasons. Cover more reasons and their evidence requirements. Today it handles item-not-received and not-as-described only.
  • Less manual data gathering. Integrate with e-commerce, fulfillment, and support tools.
  • Calibrated outcome estimates. Replace the heuristic bands with estimates from real historical outcomes, where data access and privacy allow.
  • Stronger provenance. Document-level traceability and evidence versioning.
  • Configurable policies. Let merchants set review rules and escalation thresholds without weakening the core safety guarantees.
  • Testing with real merchants. Measure review effort and error rates with actual merchants.
  • Production readiness. Security review, monitoring, and safeguards before any live PayPal support.

Every claim deserves evidence. Every decision deserves an explanation. Every action deserves consent.

Built With

Share this project:

Updates

Submission history