Verdict: Know which disputes to fight

An AI agent that handles PayPal disputes for small merchants. It scores each case, then fights, refunds or escalates inside guardrails enforced in code, not prompts.

Comment

Inspiration

When a buyer opens a PayPal dispute, a small merchant has a deadline, a pile of scattered records and one hard choice: fight or refund. Fight a case you can't win and you waste hours. Refund one you could have won and you lose the sale and the product. Most small merchants don't have anyone whose job is to make that call. We wanted an agent that makes it for them, and knows when it shouldn't.

What it does

Verdict is an AI agent for small merchants that handles PayPal disputes from start to finish.

  • Gathers evidence: order, shipment, delivery address, delivery proof and message history.
  • Scores the case: a deterministic evidence checklist combined with a language model's assessment of how winnable it is.
  • Decides: FIGHT, REFUND or ESCALATE.
  • Guardrails in code have the final say. They can only make an action safer, never riskier. Refund caps, an approval threshold, low confidence, conflicting evidence, missing evidence and expired deadlines all send the case to a human.
  • Approval queue: escalated cases show the agent's reasoning, the evidence checklist and exactly what approving will do.
  • Audit trail: every step is recorded, and the dashboard updates live.
  • Analytics: an AG Studio section lets the merchant explore disputes, deadline risk and guardrail activity, and ask a read-only analyst agent to build new charts. The analyst can never move money.

Demo (runs live against the PayPal sandbox): a sandbox buyer pays through PayPal checkout and files an item-not-received dispute. Verdict picks it up and finds no matching order or shipment record, which is exactly when an agent shouldn't guess. It rates the case at 15% win likelihood with 45% confidence and escalates instead of acting. The merchant approves an $80 refund, and PayPal shows the transaction refunded and the case closed.

How we built it

  • Backend: FastAPI and SQLAlchemy. A staged pipeline (ingest → gather → assess → decide → guard → act → log) writes an audit row for every stage with its real input, output and duration.
  • Hosting: the API and a managed PostgreSQL database run on Render, so every case, approval and audit entry survives a redeploy. SQLite is used for local development.
  • AI: openai/gpt-oss-120b, hosted on Groq, returns schema-validated JSON. Invalid output is retried, and if the model keeps failing, the case escalates instead of guessing. Conflicting evidence caps the win likelihood.
  • PayPal: OAuth, orders, evidence submission through the Disputes API, idempotent refunds (a stable request ID means a retry can never refund twice), and webhook signature verification with duplicate protection. Simulator cases are never sent to PayPal.
  • Frontend: Next.js and TypeScript. AG Grid powers the case table with expandable reasoning. AG Studio powers analytics with typed fields, relationships between evidence, audit and guardrail data, calculated fields, custom widgets and saved layouts.
  • Analyst agent: calls the model through a server-side proxy so the API key never reaches the browser. It has no tool that can approve, refund or change settings.
  • Live updates: server-sent events.
  • Quality: 499 backend tests, all passing, including a guardrail property test over 432 input combinations (action × amount × deadline × confidence × conflicts × missing evidence) proving no input makes the agent riskier than the model proposed. Plus 14 frontend unit and browser tests, and Docker and Render deploy configs.

Challenges we ran into

  • No disputes on demand. The sandbox can't generate realistic disputes on request, so we built a simulator for repeatable test cases, and every simulated case is labeled. The demo flow itself is real: a real sandbox buyer, a real dispute and a real refund.
  • Refund safety. If a refund call times out, we can't know whether PayPal processed it. Verdict marks the outcome as uncertain instead of recording a fake refund over what may be a real one, and retries reuse the same request ID.
  • Keeping the model honest. It once cited a PayPal policy that wasn't in the evidence, and it repeated conflicts the code had already found. We restricted the prompt to the supplied data and kept every decision in code.
  • PayPal auth errors. Debugging invalid_client led us to clearer error messages and a credential-check script.
  • A truthful live dashboard. Stale cells and real-time updates needed care so a closed case never looks open.

Accomplishments that we're proud of

  • A working loop on the real PayPal sandbox: purchase → dispute → AI assessment → human approval → refund → case closed.
  • Guardrails the model can't override, tested across 432 input combinations, with no input pushing the agent into a riskier action than the model proposed.
  • Honest by design: the core loop is live, and the only simulated data (generated test cases) is labeled in the UI, the audit log and the docs.
  • Self-serve analytics with a read-only AI analyst.

What we learned

  • A prompt is not a guardrail. Anything that moves money needs deterministic checks in code.
  • Design for failure. Idempotency keys and an explicit "uncertain" state matter more than the happy path.
  • Being honest about what's simulated builds more trust than a polished fake.
  • Small merchants need fewer decisions, not more dashboards.

What's next for Verdict

  • Attach delivery-proof files (photos, signed receipts) to evidence submissions, not just tracking and written notes.
  • Connect real stores and carriers instead of seeded data.
  • Support more dispute reasons, such as "not as described" and unauthorized transactions.
  • Learn from outcomes: feed won and lost cases back in to tune thresholds for each merchant.
  • Real authentication, multiple merchants, and email or chat alerts for pending approvals.

Built With

Share this project:

Updates

Submission history