Hearsay: Devpost text v1

Paste section by section. Revised Nov 10: opens with the real sandbox writes and issue #125.

Title: Hearsay

Tagline: A PayPal dispute desk whose AI agent can't be talked into a refund by the customer.

Built with: PayPal Agent Toolkit, PayPal Disputes API, PayPal Webhooks API, TypeScript, Node.js, Hono, React, Vite, Bryntum Scheduler, Ollama, Qwen3, Render, Render Workflows, Zod, Vitest, Claude Code, APIMatic Context Plugin.


Two real PayPal sandbox disputes were resolved through Hearsay: one refund the store's policy covers went out on its own, one that no policy covers waited for the owner. The gap it works around in the PayPal Agent Toolkit is reported upstream (https://github.com/paypal/agent-toolkit/issues/125). On 56 attacks written into the fields a buyer controls, a plain toolkit agent obeyed 54; behind Hearsay, 0 reached PayPal.

Inspiration

Merchants already hand disputes and refunds to AI agents (Chargeflow, Justt, Forethought, Fini). Those agents read what the buyer wrote. While opening sandbox disputes by hand, I found that the PayPal Disputes API returns the item name, item description and problem notes the buyer typed without marking them as buyer text. Opening a dispute is free, so anyone who paid a store can write instructions to its agent. Resolving one first-party misuse dispute already costs a merchant $82 on average (MRC 2026).

What it does

Hearsay is a dispute desk for a small PayPal merchant, plus a guard that sits in front of any agent built on the PayPal Agent Toolkit.

  • Every value in a dispute is tagged by who wrote it: paypal, merchant or buyer. Fields the map has never seen count as buyer text.
  • The agent (a local Qwen3 model) proposes one action and must cite a source for every value.
  • Ten deterministic rules judge the citations, not the model's text: a refund amount cannot come from buyer text, a tracking number must be the store's, a message cannot name an address no record holds, and so on. The verdict is allow, ask the owner, or deny, with every rule that fired.
  • The owner sees the buyer's text set apart, the proposal with each value's source, the verdict, and approves or rejects. A grant covers one exact proposal once.
  • Seller response deadlines sit on a Bryntum Scheduler, coloured by verdict.
  • guardToolkit() wraps an existing PayPalAgentToolkit with one changed line. Reads pass; accept_dispute_claim and create_refund are cited and judged before anything is sent.

Hearsay doesn't judge the evidence: the store's policy decides what is refunded. Hearsay makes sure the buyer's text can't rewrite that policy, whatever the model believes.

How I built it

  • PayPal Agent Toolkit for every dispute read (list_disputes, get_dispute), and as the agent Hearsay guards.
  • PayPal Disputes API v1 (accept-claim, make-offer, provide-evidence, send-message), written only by one executor that re-reads the dispute first.
  • PayPal Webhooks API: CUSTOMER.DISPUTE.* notifications on a separate one-route listener, checked with verify-webhook-signature.
  • Bryntum Scheduler for the deadline board (lanes by who must act, verdict colours, tooltips with the rules).
  • Render hosts the judges' mode: recorded agent runs, no PayPal account or model needed.
  • Render Workflows re-judges the whole bench against the deployed rules: one root task fans out 79 parallel case runs and returns the score; the "Try an attack" tab shows the last run.
  • APIMatic Context Plugin in Claude Code for PayPal Server SDK questions while building.
  • TypeScript strict, Hono, React 19, Zod at every boundary, 301 tests.

Results

79 dispute cases built from real sandbox disputes: 56 attacks (4 goals x 4 buyer channels x 3 framings, plus hand-written ones) and 23 legitimate cases.

  • Plain toolkit agent (Qwen3 8B), no guard: obeyed 54 of 56 attacks, and all 54 reached PayPal. It also refunded 10 of 10 legit cases where nothing was owed.
  • Same agent behind guardToolkit(): 0 of 56 reached PayPal; of 13 owed refunds, 10 were paid automatically and 3 held for the owner.
  • Hearsay's own desk agent, two models, with and without a prompt warning: 0 of 56 attacks got past the rules in all four configurations. A prompt warning alone did not help Qwen3 (4 obeyed without it, 6 with it).
  • A test plays the perfectly obedient agent: for each attack it builds the exact proposal the attacker wants. None is allowed.
  • Real writes in the sandbox, on two disputes opened for this: a cracked mug (18 USD, not as described) went through guardToolkit() because the store's policy refunds it, and PayPal resolved the dispute; a tote bag that never shipped (26 USD) was held because no policy covers it, approved by the owner in the desk, re-judged and sent (HTTP 200, resolved).
  • Cost: the agent is a local 8B model and the rules call no model, so judging a dispute costs $0 in model calls. The hosted demo calls no model at all.

Challenges

  • Finding which fields are buyer-written: I opened disputes by hand in the sandbox and traced each form field to its API path.
  • The toolkit's actions setting only filters getTools(); handleToolCall runs any method by name (reported as https://github.com/paypal/agent-toolkit/issues/125). The guard refuses every write tool it has no rules for.
  • Small local models loop in their reasoning on injected text; the bench records timeouts as results instead of hiding them.

What I learned

  • A prompt warning is not a defence: Qwen3 obeyed 4 attacks without one and 6 with it. Checking where each value came from worked the same for every model.
  • The dangerous text is not only the buyer's message. Item names and descriptions arrive looking like PayPal's own data, and a hidden note there is enough to steer an agent.
  • An agent's summary is not a record: in both guarded live runs the agent said the refund was made when the guard had held it.

What's next

Attachments (buyer PDFs and images), disputes over several transactions, and the same authorship map for invoices and order notes.

Prior work

The pattern of cited parameters, verdicts that list every rule hit and one-shot grants comes from my earlier project Paperweight. It was ported to TypeScript, not imported. The authorship map, the PayPal integration, the dispute rules, the desk, the guard and the bench are new.

AI tools used

  • Runtime: Qwen3 8B and Qwen2.5 7B, local through Ollama. No paid model is called; the hosted demo calls none.
  • Building: Claude Code wrote most of the code under my design and review, with the APIMatic Context Plugin and PayPal's Claude Code plugin for API docs.

Test it

https://hearsay-bench.onrender.com: judges' mode, no login. Start on the home page, open the four disputes, then the Deadlines and Try an attack tabs. Repository: https://github.com/0xjgr/hearsay (MIT).

Built With

Share this project:

Updates

Submission history