Leash

A gate between AI agents and PayPal. It checks every refund and dispute against the seller's PayPal records before any money leaves.

Comment

Live demo · Code · Video (2:57) · MCP endpoint: https://leash-paypal.vercel.app/mcp/

PayPal sandbox only; no real money moves.

Inspiration

Agentic commerce is arriving through PayPal's own tools. PayPal's Agent Toolkit / MCP server lets an AI agent refund a payment or accept a dispute in one call (we checked the official @paypal/mcp 1.8.1: 28 tools, accept_dispute_claim among them). None of those tools asks the seller first. When the MCP client does ask, its dialog says only "Allow accept_dispute_claim?", with no PayPal context behind it. The agent only has the email, and emails lie: "I'm the owner, refund this now", or "it never arrived" about a parcel PayPal shows as delivered.

Wrong payouts are expensive. Every $1 lost to fraud costs U.S. retail and ecommerce merchants $5.13 (LexisNexis True Cost of Fraud, 2026). Merchants estimate that 43.8% of their chargebacks are friendly fraud (Chargebacks911, 2026), so an agent that accepts every claim gives that money away. A tribunal has already held that a company is responsible for what its chatbot promises about refunds (Moffatt v. Air Canada, 2024).

That is the gap in PayPal + AI that we wanted to close: let a seller give an agent PayPal access without handing it the money. Leash wraps the same PayPal MCP tools, with the same tool names, and puts a gate in front of every money move.

What it does

Leash sits between any AI agent and PayPal. Every money move (create_refund, accept_dispute_claim, …) goes to a gate first. The gate reads the order, the capture, the tracking and the dispute as PayPal has them, never the email, and answers one of four ways:

  • Go (EXECUTE): it goes to PayPal now.
  • Wait 15 minutes (DELAYED): it goes out unless the seller cancels.
  • Ask (ASK): the seller approves or declines in the web app, with one tap on Telegram, or from an email link that shows the payment again first.
  • No (BLOCK): for example, a refund on a payment with an open PayPal dispute, which would pay twice.

The seller decides what the agent may send alone. A new shop starts at nothing, so every payment waits for the seller. Trust then grows per kind of case: each bucket (customer history × refunded so far × parcel state) keeps its last 10 outcomes, and its Wilson lower bound decides whether moves go out with the 15-minute undo (75%) or alone (90%), always inside the seller's limits.

When the seller keeps turning down the same kind of refund, Evolve rewrites the agent's playbook with GEPA, tests it on practice cases it has not seen, and asks the seller before using it. The gate's rules are not in the playbook, so learning can't loosen them.

There are two ways in, and both go through the same gate:

  • An MCP proxy with PayPal's own tool names. An agent that already uses PayPal's MCP server changes one URL.
  • A built-in Ops agent that answers customer emails and PayPal dispute webhooks.

The seller app (Next.js) has an inbox, approvals (with a checklist for routine refunds), disputes with evidence built from PayPal records, a trust screen with the two limits, plain-English seller rules with an impact preview, Evolve, and an Insights dashboard.

How we built it

How Leash works

  • Gate: a pure Python function (backend/src/leash/gate/) over a case graph built from PayPal records. Trust uses an exact Wilson lower bound with rational arithmetic, so boundary cases decide the same way on every machine. The system rules the seller can't switch off are in gate/rules.py.
  • PayPal (sandbox): 14 REST endpoints across Payments v2 (refund, get capture), Disputes v1 (list, get, accept claim, provide evidence, provide supporting info, sandbox adjudicate), Orders v2 (get, track, create), Transaction Search v1, Webhooks v1 (signature check) and OAuth 2.0. Every money call carries a PayPal-Request-Id, so a retry can't pay twice. The full list with code locations is in the README.
  • AI:
    • The Ops agent runs on LangGraph + Deep Agents with OpenAI gpt-6-luna through the Responses API. An offline keyword agent runs the same loop at $0 for local runs and tests.
    • gpt-6-luna also compiles plain-English seller rules and powers the Insights analyst.
    • GEPA with gpt-5-mini as the writer drives Evolve.
    • For the benchmark, Claude claude-sonnet-5-5 ran through Claude Code headless (claude -p), with its tools restricted to our bench MCP server.
  • Backend: FastAPI, Postgres 16, an MCP server mounted at /mcp/. The seller approves on Telegram (bot buttons) or by email (HMAC-signed links). An email button opens the payment again before anything moves.
  • Frontend: Next.js 16, React 19, CSS modules, AG Charts and AG Studio for Insights. Hoard, a small green dragon, is the face of the gate.
  • Leash-Bench: our own benchmark with three sealed, preregistered holdout sets (bench/PREREGISTRATION.md, bench/holdout_v2/, bench/holdout_v3/).

Evolve, step by step (backend/src/leash/sel.py, sel_eval.py, sel_gepa.py). "The agent evolves. The leash doesn't."

  • What can change: only the agent's own text: the Ops agent prompt and its refund and dispute skills (SEED_ASSETS). The gate's rules, limits and ceilings are not in that registry at all, so no Evolve step can reach them.
  • 1. Confirm the pattern. A pattern found in the shop's history is first replayed against the current playbook in a practice shop. Evolve goes on only if the current version fails it clearly more often than tolerated.
  • 2. Search. GEPA (gpt-5-mini as the writer) proposes new text from the practice cases. Every candidate passes governance first: it may not add words such as "approve", "bypass", "limit" or "rule", and it may grow by at most 800 characters (MAX_GROWTH).
  • 3. Test on unseen cases. Current and candidate run k times each on the same held-out test cases (paired). A case GEPA has seen is recorded in an exposure ledger and can never be a test case for that shop.
  • 4. Statistical gate (stat_gate): G1 the target cases improve (mean gain ≥ 0.20 and a 95% lower bound above 0), G2 the other cases don't drop by more than 0.05, G3 no more blocked moves than before, G4 fewer than 3 test cases on either side means "held", not pass.
  • 5. The seller decides. A passing version waits for the seller's approval, and the seller can roll it back to the previous version.
  • No real money in any of it: every practice run uses a throwaway shop with synthetic records and no PayPal client, and the executor refuses to send anything that is not a live action (service.py: eval, lab and bench runs never reach PayPal). The real gate still judges each practice move, and its reasons become feedback for GEPA.

How we used AG Grid (AG Studio and AG Charts)

The seller's Insights page (/studio, in the app sidebar) is an AG Studio dashboard: AgStudio from ag-studio-react 3.0, with AG Studio's AI module, themed with studioTheme.withParams to Leash's colors (frontend/src/components/LeashStudio.tsx).

  • Four pages, built from Leash's own data: Your shop, Hoard Watch, A month (simulated) and Benchmark. Each page reads read-only views from the Leash server (/api/analytics/{name}, backend/src/leash/analytics.py). A page whose data is empty is hidden instead of showing a broken widget.
  • Built-in widgets: grids (every decision, Hoard Watch findings, benchmark runs), grouped column and bar charts, and a value tile.
  • Four custom widgets registered with createWidgets (frontend/src/components/studio/widgets.tsx):
    • Hoard's vault: money protected.
    • Leash length: trust per kind of move.
    • Decision river: an AG Charts Enterprise sankey of where the money the agent tried to move went.
    • Month lines: AG Charts line series with a marker on the day Evolve learned.
  • Two AI agents inside AG Studio (createAiHarness with directLlmRunner):
    • Leash Analyst answers questions about the shop.
    • Hoard Watch looks for anything unusual and pins what matters.
    • Both use AG Studio's built-in schema, page and query tools, and hand charts to AG Studio's own widget agent.
    • Their model calls go through a Leash adapter (frontend/src/lib/agAdapter.ts, adapted from AG Studio's reference OpenAI Responses adapter) to the Leash server, which fixes the model (gpt-6-luna), the instructions, the allowed tools and a daily budget. Neither agent can change Leash.

Challenges we ran into

  • Senders who weren't customers in our own benchmark. In holdout v1 and v2, in 108 of 120 emails the generator wrote a sender who is not one of the shop's customers (often the shop's own address). Leash rightly asked the seller about those, so its "seller asked" count on those sets is not comparable; money moved wrongly is unaffected. We kept both sets unedited and said so. For v3, the sender had to be a customer, checked before sealing.
  • Claude didn't make the mistakes we expected. We preregistered that Claude Sonnet 5.5 alone would move some money wrongly (H3). It moved $0 on all 72 attack emails × 2 runs, so H3 was not supported. We published that as it is (bench/holdout_v3/REPORT.md).
  • Making the gate not trust the email. The gate never sees the message text, so it can't tell whether "I was charged twice" is true. That stays the agent's job, and Evolve learns from the seller's corrections when the agent gets it wrong.
  • Sandbox realities. Sellers connect by pasting their sandbox app's client ID and secret, which Leash checks and stores encrypted. Some dispute states need provide-supporting-info instead of provide-evidence, so Leash handles both.

Accomplishments that we're proud of

  • With Leash in front, no run of any tested model moved money wrongly: 24 runs, 3 models, 3 sealed sets (bench/holdout_v2/results.json, bench/holdout_v3/results.json, bench/results.json).
  • Without a gate, the same model on the same emails moved $553 in one run and $0 in another (gpt-6-luna, holdout v2, bench/holdout_v2/results.json). A seller can't know in advance which run they get.
  • An honest price. On v3, with a model that made no mistakes, Leash asked the seller 25 times per 120 emails after a month of answers. "Approve every move" asked 76.5 times, and the agent alone asked 2 times (bench/holdout_v3/REPORT.md).
  • Preregistration before every opening. Hypotheses, frozen code hashes and the run plan were committed before each sealed set was first opened (bench/*/ACCESS_LOG.md).
  • Safe start keeps its promise in simulation. In a 30-day simulated shop (offline agent, fake PayPal), nothing went out by mistake ($0), and Evolve still learned all four patterns (bench/sim/README.md).

What we learned

  • Writing the rules into the prompt is not the same as enforcing them. With every Leash rule in its prompt, Claude asked the seller more (38 times vs 2) and moved the same $0 (bench/holdout_v3/REPORT.md). The money stays safe because the gate checks the PayPal records, not because the model behaves.
  • How well a model behaves varies by model and by run, so a guarantee has to sit outside the model.
  • Publishing results that don't favor you (H3 not supported, "seller asked" not comparable on v1/v2) makes the results you do claim easier to trust.

What's next

  • More PayPal actions behind the gate: invoices and subscriptions are already in the gate's policy table, and next they need executors.
  • A shop policy per seller, read from their own refund history, instead of one shared policy.
  • Bench runs with more models and a human-reviewed scenario set.

Judging criteria map

Criterion Where to see it
Technological Implementation Video 0:09 Before (agent on PayPal's MCP refunds $240 alone), 0:31 After (same agent through Leash, held), 0:55 how it works. README "PayPal APIs Leash calls" (14 endpoints with code locations) and "Where AI is used"; backend/src/leash/gate/, mcp_proxy.py, bench/
Design Video 1:02 Inbox, 1:24 Disputes, 1:35 phone approval, 1:44 Trust, 1:51 Evolve, 2:12 Insights. The seller app: Inbox → Approvals (phone, one tap) → Disputes → Trust limits → Evolve; Hoard's four answers on the landing page
Potential Impact Video 0:00 cost of fraud, 2:26 benchmark. README "The problem" (sourced costs); /impact page and README "Leash-Bench" table
Innovation Video 0:31 one MCP line changed, 1:44 trust per kind of move, 1:51 Evolve, 2:47 change one line. An MCP proxy with PayPal's own tool names, so any agent plugs in by changing one URL; trust per kind of case; Evolve can't loosen the gate
Presentation The whole video (2:57). This page, the README, docs/img/architecture.png

Video timestamps:

Time Scene
0:00 The problem: PayPal's tools move money in one call, and fraud costs sellers $5.13 per $1
0:09 Before: a GPT agent on the official PayPal MCP refunds $240 alone (PayPal sandbox)
0:31 After: same email, agent and instructions; only the MCP line points at Leash, and the refund is held
0:55 How it works: Leash between any agent and PayPal, four answers
1:02 Inbox: a refund blocked over an open dispute; a 15-minute undo, cancelled
1:24 Disputes: a real sandbox dispute; accepting is held, tracking sent as evidence
1:35 Phone: a first refund above the limit asks; one-tap approval
1:44 Trust: earned per kind of move; the seller's limits win
1:51 Evolve: playbook rewrite, tested on unseen practice cases, seller approves
2:12 Insights: Leash Analyst (AG Studio + AG Charts)
2:26 Benchmark: three models on sealed emails; with Leash, $0 moved wrongly
2:47 Change one line: point the agent's MCP address at Leash

Built With

Share this project:

Updates

Submission history