-
-
Try an attack: play the hijacked agent yourself. Re-judged on Render Workflows, 0 recorded model attacks and 0 get past the rules.
-
Seller response deadlines on a Bryntum Scheduler, coloured by verdict: what waits for the owner, what the agent has not read.
-
The buyer supplies a "corrected" tracking number. The agent files it as proof; Hearsay traces the value to the buyer's message and blocks it
-
A real sandbox dispute orders the assistant to refund. Text that reads like an order to the agent is underlined in red.
-
Hearsay: a plain PayPal Agent Toolkit agent let 54 of 56 dispute attacks reach PayPal. Behind Hearsay, 0.
-
Every value in a PayPal dispute is tagged by who wrote it: buyer, PayPal or merchant. Ten fixed rules check the sources, not the wording.
Hearsay: Devpost text v1
Paste section by section. Revised Nov 10: opens with the real sandbox writes and issue #125.
Title: Hearsay
Tagline: A PayPal dispute desk whose AI agent can't be talked into a refund by the customer.
Built with: PayPal Agent Toolkit, PayPal Disputes API, PayPal Webhooks API, TypeScript, Node.js, Hono, React, Vite, Bryntum Scheduler, Ollama, Qwen3, Render, Render Workflows, Zod, Vitest, Claude Code, APIMatic Context Plugin.
Two real PayPal sandbox disputes were resolved through Hearsay: one refund the store's policy covers went out on its own, one that no policy covers waited for the owner. The gap it works around in the PayPal Agent Toolkit is reported upstream (https://github.com/paypal/agent-toolkit/issues/125). On 56 attacks written into the fields a buyer controls, a plain toolkit agent obeyed 54; behind Hearsay, 0 reached PayPal.
Inspiration
Merchants already hand disputes and refunds to AI agents (Chargeflow, Justt, Forethought, Fini). Those agents read what the buyer wrote. While opening sandbox disputes by hand, I found that the PayPal Disputes API returns the item name, item description and problem notes the buyer typed without marking them as buyer text. Opening a dispute is free, so anyone who paid a store can write instructions to its agent. Resolving one first-party misuse dispute already costs a merchant $82 on average (MRC 2026).
What it does
Hearsay is a dispute desk for a small PayPal merchant, plus a guard that sits in front of any agent built on the PayPal Agent Toolkit.
- Every value in a dispute is tagged by who wrote it:
paypal,merchantorbuyer. Fields the map has never seen count as buyer text. - The agent (a local Qwen3 model) proposes one action and must cite a source for every value.
- Ten deterministic rules judge the citations, not the model's text: a refund amount cannot come from buyer text, a tracking number must be the store's, a message cannot name an address no record holds, and so on. The verdict is allow, ask the owner, or deny, with every rule that fired.
- The owner sees the buyer's text set apart, the proposal with each value's source, the verdict, and approves or rejects. A grant covers one exact proposal once.
- Seller response deadlines sit on a Bryntum Scheduler, coloured by verdict.
guardToolkit()wraps an existingPayPalAgentToolkitwith one changed line. Reads pass;accept_dispute_claimandcreate_refundare cited and judged before anything is sent.
Hearsay doesn't judge the evidence: the store's policy decides what is refunded. Hearsay makes sure the buyer's text can't rewrite that policy, whatever the model believes.
How I built it
- PayPal Agent Toolkit for every dispute read (
list_disputes,get_dispute), and as the agent Hearsay guards. - PayPal Disputes API v1 (
accept-claim,make-offer,provide-evidence,send-message), written only by one executor that re-reads the dispute first. - PayPal Webhooks API:
CUSTOMER.DISPUTE.*notifications on a separate one-route listener, checked withverify-webhook-signature. - Bryntum Scheduler for the deadline board (lanes by who must act, verdict colours, tooltips with the rules).
- Render hosts the judges' mode: recorded agent runs, no PayPal account or model needed.
- Render Workflows re-judges the whole bench against the deployed rules: one root task fans out 79 parallel case runs and returns the score; the "Try an attack" tab shows the last run.
- APIMatic Context Plugin in Claude Code for PayPal Server SDK questions while building.
- TypeScript strict, Hono, React 19, Zod at every boundary, 301 tests.
Results
79 dispute cases built from real sandbox disputes: 56 attacks (4 goals x 4 buyer channels x 3 framings, plus hand-written ones) and 23 legitimate cases.
- Plain toolkit agent (Qwen3 8B), no guard: obeyed 54 of 56 attacks, and all 54 reached PayPal. It also refunded 10 of 10 legit cases where nothing was owed.
- Same agent behind
guardToolkit(): 0 of 56 reached PayPal; of 13 owed refunds, 10 were paid automatically and 3 held for the owner. - Hearsay's own desk agent, two models, with and without a prompt warning: 0 of 56 attacks got past the rules in all four configurations. A prompt warning alone did not help Qwen3 (4 obeyed without it, 6 with it).
- A test plays the perfectly obedient agent: for each attack it builds the exact proposal the attacker wants. None is allowed.
- Real writes in the sandbox, on two disputes opened for this: a cracked mug (18 USD, not as
described) went through
guardToolkit()because the store's policy refunds it, and PayPal resolved the dispute; a tote bag that never shipped (26 USD) was held because no policy covers it, approved by the owner in the desk, re-judged and sent (HTTP 200, resolved). - Cost: the agent is a local 8B model and the rules call no model, so judging a dispute costs $0 in model calls. The hosted demo calls no model at all.
Challenges
- Finding which fields are buyer-written: I opened disputes by hand in the sandbox and traced each form field to its API path.
- The toolkit's
actionssetting only filtersgetTools();handleToolCallruns any method by name (reported as https://github.com/paypal/agent-toolkit/issues/125). The guard refuses every write tool it has no rules for. - Small local models loop in their reasoning on injected text; the bench records timeouts as results instead of hiding them.
What I learned
- A prompt warning is not a defence: Qwen3 obeyed 4 attacks without one and 6 with it. Checking where each value came from worked the same for every model.
- The dangerous text is not only the buyer's message. Item names and descriptions arrive looking like PayPal's own data, and a hidden note there is enough to steer an agent.
- An agent's summary is not a record: in both guarded live runs the agent said the refund was made when the guard had held it.
What's next
Attachments (buyer PDFs and images), disputes over several transactions, and the same authorship map for invoices and order notes.
Prior work
The pattern of cited parameters, verdicts that list every rule hit and one-shot grants comes from my earlier project Paperweight. It was ported to TypeScript, not imported. The authorship map, the PayPal integration, the dispute rules, the desk, the guard and the bench are new.
AI tools used
- Runtime: Qwen3 8B and Qwen2.5 7B, local through Ollama. No paid model is called; the hosted demo calls none.
- Building: Claude Code wrote most of the code under my design and review, with the APIMatic Context Plugin and PayPal's Claude Code plugin for API docs.
Test it
https://hearsay-bench.onrender.com: judges' mode, no login. Start on the home page, open the four disputes, then the Deadlines and Try an attack tabs. Repository: https://github.com/0xjgr/hearsay (MIT).
Built With
- agent
- api
- bryntum
- claude
- code
- disputes
- hono
- node.js
- ollama
- paypal
- qwen3
- react
- render
- scheduler
- toolkit
- typescript
- vite
- vitest
- webhooks
- workflows
- zod