Inspiration
A customer writes: "I was charged but never got a confirmation. Refund me."
For a small online merchant, that one message starts a hunt. Was the payment actually captured? Was it already refunded? Is there an open dispute? The answers sit across orders, captures, refunds and disputes, and the merchant has to make a money-moving decision under pressure. Refunding twice, refunding during an open dispute, or refunding on an unverified claim are all easy mistakes, and each one costs real money.
We didn't want another chatbot that sounds confident. We wanted an AI that investigates like a detective, argues the case from both sides, and cannot move money on its own.
What it does
1. Evidence-first investigation. The backend, not the AI, collects live PayPal evidence (order, captures, refunds, disputes, verified webhooks) through PayPal's MCP server, and numbers every item (EV-001, EV-002, …). Code then computes a deterministic FactSheet in integer minor units, so no floating-point error ever touches money:
$$ \text{remaining} = \text{captured} - \text{completed refunds} - \text{pending refunds} $$
A Gemini investigator writes the report, and every finding must cite evidence. Code deletes anything uncited, and the customer's complaint is labelled an allegation, never proof.
2. An AI courtroom for every refund. Every money-moving decision goes to an Adversarial Decision Council:
- an Advocate argues 100% for the refund and a Challenger argues 100% against it, in parallel, blind to each other, on different models;
- code strikes any argument that cites evidence that doesn't exist;
- an impartial Judge rules;
- deterministic guardrails, recomputed from live PayPal data, have the last word, and they can only make the verdict stricter:
$$ \text{final verdict} = \max(\text{Judge verdict}, \text{guardrail minimum}) $$
where verdicts are ordered from least to most strict: APPROVE < APPROVE_MODIFIED < NEEDS_MORE_INFO < ESCALATE_TO_HUMAN < REJECT. Guardrails can only move the verdict to the right.
3. The merchant delivers the final verdict. A refund needs a passing council verdict, a live re-check of eligibility, a lock, a one-time idempotency key, and the merchant's explicit confirmation on a button showing the exact amount. Only then does it run create_refund, exactly once. The result is verified with PayPal's own get_refund, and every step is written to an audit trail.
Around that core: a transactions explorer, disputes with a council-gated "accept claim", Insights whose AI observations must cite the metric they're based on, a Settings page with live MCP and AI health, and a clearly labelled Demo mode when no credentials are set.
How we built it
- PayPal: the backend spawns and owns the
@paypal/mcpserver over stdio and discovers its 28 tools. It allowlists only 9 of them and validates every call's arguments against the tool's own schema.- Reads:
list_transactions,get_order,get_refund,list_disputes,get_dispute. - Approval-gated writes:
create_refund,accept_dispute_claim. - OAuth client-credential tokens refresh automatically. A REST fallback is used only if MCP is down, and webhook signatures are verified with PayPal's API.
- Reads:
- AI: Google Gemini through its OpenAI-compatible endpoint, with JSON-schema structured output validated by Zod. Each role runs on its own model with its own fallback:
| Role | Model |
|---|---|
| Investigator | gemini-3.5-flash |
| Advocate | gemini-3.5-flash |
| Challenger | gemini-3-flash-preview |
| Judge | gemini-3.6-flash |
- Stack: Node 22, TypeScript, Express 5, SQLite, React 19, Vite, Tailwind 4 and TanStack Query, with 151 server and 26 client tests (Vitest). Deployed on Render as a single web service.
- Built with Claude Code (Claude Opus 5.5) as our AI pair programmer for design, implementation, tests and docs.
Challenges we ran into
- Refunds are invisible in
get_order. After our first real Sandbox refund, the FactSheet still said "0.00 refunded", because PayPal's Orders API doesn't list refunds. We now discover refunds throughget_refundand PayPal'suplinks, and we fail safe: if a capture saysPARTIALLY_REFUNDEDbut we can't see the refunds, further refunds are blocked instead of guessed. - No idempotency key on MCP
create_refund. We built our own: a unique key per approval, locks, and reconciliation for unknown outcomes.create_refundis never retried blindly. - AI rate limits mid-trial. Free-tier quotas caused a timed-out report and a hung council call. We added per-role fallback models (the two parallel agents never share one) and per-attempt timeouts. When a side fails, the council escalates to a human instead of letting the Judge hear only one side.
- Sandbox realities: a buyer from another country was declined in another currency, declined capture attempts stay on the order, and Transaction Search lags behind. Each case is handled explicitly and shown honestly in the UI.
Accomplishments that we're proud of
- A real end-to-end Sandbox refund:
5YE09135FX432004L, USD 20.00. Investigation → council (APPROVE, confidence 0.95) → merchant approval →create_refund→get_refundCOMPLETED, with 13 audit entries. - It caught a double refund live. Asked to refund the full 64.00 after that 20.00 refund, the Challenger cited the completed refund and the Judge rejected it. Replaying the old approval returned
409 ALREADY_EXECUTED. - It survives prompt injection. In a test where the Judge obeys an instruction injected into the complaint, the guardrails still block the refund.
What we learned
LLMs are excellent at weighing evidence and terrible at being the source of truth for money. The design that works splits the job three ways:
$$ \text{facts (PayPal + code)} \rightarrow \text{judgement (AI council)} \rightarrow \text{decision (merchant)} $$
Making two agents argue against each other surfaced risks a single assistant would have politely skipped. And stricter-only guardrails mean the worst a confused model can do is ask a human.
What's next for Refund Detective
- Merchant accounts with roles (investigator vs. approver).
- Submitting dispute evidence, once PayPal exposes an API or MCP tool for it.
- Re-investigating automatically when a webhook reports a new dispute.
- Durable Postgres storage and production AI quotas.
Built With
- better-sqlite3
- claude-code
- express.js
- google-gemini
- model-context-protocol
- node.js
- paypal
- paypal-mcp
- paypal-rest-api
- paypal-sandbox
- react
- recharts
- render
- sqlite
- tailwindcss
- tanstack-query
- typescript
- vite
- vitest
- zod