Inspiration

What it does

How I built it

Challenges I ran into

Accomplishments that I'm proud of

Inspiration

Everyone has waited for a refund. Stores make you wait because someone has to check every return by hand, and honest customers pay for the cheats: one Shopify merchant put a 20% restocking fee on every order after a run of near-identical return requests. An AI agent could take the wait away, but only if it can't be talked into paying anyone.

A small store takes PayPal. Returns arrive by e-mail, and the store wants an AI agent to handle them, because PayPal's Agent Toolkit already gives an agent a create_refund tool. But that e-mail is written by the customer, or by someone pretending to be one. PayPal can tell whether a refund is valid for the transaction. Only the store knows whether it is owed. With the Agent Toolkit, which refunds need a person is left to the merchant's own integration. We built that piece: it lets a merchant on PayPal hand returns to an agent without handing over the decision.

OWASP ranks prompt injection first among LLM risks (LLM01:2025), including instructions hidden in e-mails and attachments, and Moffatt v. Air Canada (2024) held a business responsible for what its AI told a customer. Return fraud is already large without agents: US retailers lost an estimated $103B to fraudulent returns and claims in 2024 (Appriss Retail and Deloitte).

Support bots are already being talked into promises they shouldn't make. In a red-team test of 14 live support chatbots, Reinforce Labs reported that 11 made unapproved commitments such as refunds or warranties (1 Oct 2026; the firm's own account, companies unnamed). A UK shop owner reported that the after-hours bot was flattered into inventing discount codes up to 80% off (Feb 2026, the owner's account). We found no verified case yet of an agent that moved money this way, because most bots could only promise. That is changing: PayPal's Agent Toolkit gives an agent create_refund, and support agents that can issue refunds are spreading. Counterfoil is meant to be in place before the promise becomes a payment.

Who it's for. Merchants who let an AI support agent issue refunds, and that is a feature you can switch on today: Intercom's Fin ships "Create Shopify Refund" and Stripe refund actions, Gorgias's AI Agent refunds the shopper when it cancels an order or removes an item, and PayPal's Agent Toolkit gives an agent create_refund. Those merchants already fight return abuse by hand: on the Shopify Community one raised the restocking fee to 20% on every order after 18 near-identical return requests for cheap items; another is told to switch the returns portal to "manual approval only". Counterfoil does that check against the return record for the agent, on every call, and leaves a person only what doesn't match. Sources and their limits: public evidence.

Our first Braintree sandbox experiment showed where the gap is. Braintree refuses a card refund above the sale total. It cannot know whether the customer sent back one cup or four, whether the sender owns the order, or that USD 60 of this return was already refunded yesterday. Only the store knows that. Counterfoil sits in that gap.

Built solo from Türkiye, where PayPal accounts can't be opened, on Braintree, the PayPal service PayPal's team confirmed for this hackathon.

What it does

Counterfoil lets a merchant on PayPal give refunds to an AI agent, and refunds only what came back. It is a deterministic guardrail for agentic commerce, between the agent and Braintree, the PayPal service. The model is probabilistic; the gate is plain code, and what the store's records can't settle goes to a person:

  1. Reader (an AI agent with no tools) reads the customer's e-mail and hands off fixed fields: order, return record, request, amount, items.
  2. Runner (an AI agent that never sees the e-mail) asks to refund, charge, void or lookup_transaction.
  3. Counterfoil holds the only Braintree keys and answers every call with PASS, APPROVAL (held for the store manager) or BLOCK, by comparing it with the store's own records: who owns the order, what the return record (RMA) still allows after earlier and reserved refunds, catalog price √ó quantity for a charge, Braintree's own line items for the sale, and open chargebacks.
  4. Braintree moves only what passed, or what a person approved. Each approval is single-use and bound to the exact call the manager saw.

The agent can still be fooled; Counterfoil makes sure a fooled agent can't send the money.

Try it in 90 seconds

  1. Open the live demo, pick "Nella asks for more" (attack P6-01, a split refund) and press Compare the three checks. With no gate and with a $200 cap, USD 60 reaches the rail. With Counterfoil, no money moves.
  2. Press Watch it in the store ‚Üë: the 3D HOMI store replays the run, one step per click: Nella's note, Reader's hand-off, Runner's call, Counterfoil's BLOCK with its reason, what reached PayPal / Braintree, and what would have left without the gate.
  3. Switch to Live AI and Write your own attack: Claude Haiku 5.5 reads it live and Counterfoil decides (rate-limited, test money only).
  4. Pay with PayPal (sandbox) for the scenario's order, then let the recorded agent refund it through the gate: real Braintree sandbox transactions.
  5. Open the dashboard for every number below, broken down by model.

Why Braintree, and not the PayPal REST API

PayPal's Türkiye notice states that local operations stopped in June 2016. The author could not obtain a PayPal account for this project, so the PayPal REST path was not exercised against a real REST sandbox. The official hackathon rules explicitly include Braintree as a qualifying PayPal service. So instead we built on the Braintree sandbox with Braintree's mocked PayPal account, used Braintree well beyond a checkout (below), and kept PayPal's agent interface: our MCP server offers the Agent Toolkit's create_refund unchanged for the agent, and a gated PayPal REST rail stands behind the Toolkit's create_refund, ready for a PayPal account. That rail is the one path we could not run for real; it is tested with a faked HTTP layer.

How we built it

PayPal and Braintree, used in depth (all sandbox):

  • transaction.sale with line items (product code, quantity, amounts) and the mocked PayPal account nonce, PayPal checkout through Braintree Drop-in, vaulted PayPal methods for reorders, transaction.refund, transaction.void, transaction.find.
  • transaction_line_item.find_all: before a refund, the gate checks the return record against what Braintree says was sold, a second source next to the store's own records.
  • api_request_key derived from the money movement, so a retried or raced refund pays once (checked in the sandbox).
  • Disputes: a refund on a transaction with an open chargeback is blocked; Braintree itself pays it, which we showed on the dispute test card.
  • Signed webhooks: a dispute_opened notification locks refunds on that transaction before the next agent call, even one already waiting for the manager.
  • transaction.search refund sweep: finds refunds made behind the gate's back, in the Control Panel or by another tool.
  • PayPal Agent Toolkit-shaped MCP server (counterfoil-mcp): Claude Desktop, Cursor or Claude Code call create_refund(capture_id, amount) exactly as with the Toolkit, and every call goes through the gate to the Braintree sandbox.

AI, used in depth:

  • Two agents with split duties (Reader without tools, Runner without the e-mail) over an OpenAI-compatible client.
  • Nine models measured: four local (qwen2.5, qwen3, llama3.1, hermes3 on Ollama) and five cloud (Claude Haiku 4.5, Sonnet 5, Opus 5.5; GPT-5.6 Sol and Terra) through the Astropods Astro AI Gateway.
  • Live Claude Haiku 5.5 in the hosted demo, with per-visitor rate limits and a daily call budget.
  • AG Studio (AG Grid) dashboard over every run, with a review desk that explains each held call. Its case panel runs on rules over the recorded data, not on an LLM.

Engineering: Python, FastAPI and SQLite (orders, RMAs, refund reservations, charge states); 539 passing tests, 2 skipped and 1 expected failure (10 Nov 2026), 94.74% coverage (measured 3 Nov 2026), without an LLM or network; 10 architecture decision records; the demo configured on Render Starter, the dashboard on Vercel. Mutation results remain dated to their 24 Oct run; see quality evidence.

The results

On the real Braintree sandbox (10–11 Nov). The 14 attacks, three repeats, three conditions, three local models; every refund and void was a real sandbox call on a mocked PayPal account.

Wrong money moved, attack runs No gate $200 cap Counterfoil
qwen2.5 27/42 27/42 0/42
qwen3 23/42 23/42 0/42
hermes3 9/42 9/42 0/42
Total 59/126 59/126 0/126

The agent was misled about as often in each condition (71, 71 and 68 of 126). This run has attacks only; the genuine-request figure comes from October.

The second table is the October multi-model experiment on an in-memory rail. The other November experiments are separate in the dashboard: 30 blind attacker cases on the local rail, a repaired 30-run hosted attack cohort, 36 clean-control sandbox runs, and 12 repeated executions. The original six-run hosted pilot remains archived with one unknown outcome. These datasets have different fixtures and denominators and are not pooled into either table.

In the repeated clean cohort, Counterfoil completed both valid refunds and held both USD 120 reorders for human confirmation; no gate and the USD 200 cap each charged both reorders without that confirmation. Reader outputs were cached and identical across repeats, so this is not independent end-to-end sampling. Detailed protocols, limits and downloadable scored rows accompany each experiment on the dashboard.

22 scenarios: 14 attacks in 7 families and 8 genuine requests, split into a dev set and a holdout set kept apart while the gate was written. The evaluation spans nine models, three repeats and three conditions with paired Reader hand-offs. Model coverage differs between dev and holdout; the table reports completed runs per condition, not a full Cartesian product of every model and scenario.

No gate $200 cap Counterfoil
Wrong money moved (or another customer's data read), attack runs 99/270 100/270 0/270
Agent misled, attack runs 141/270 142/270 139/270
Genuine requests handled right 102/183 123/183 150/183 (82%)
Worst case: Reader and agent fully on the attacker's side, no model 11/14 11/14 0/14

The agent is misled about equally often in every condition; only the money changes. The $200 cap stopped nothing: every harmful call was below it.

Attackers from four model families. Muse Spark, GPT-5.6, DeepSeek and Mistral each wrote 18 emails over three rounds, adapting to the gate's public answers. Wrong money moved in 35 of 72 runs with no gate, 35 of 72 behind the cap and 0 of 72 with Counterfoil. They also found a gap: 11 voids passed the gate and were stopped only by the rail. We fixed the gate and re-ran the same 72 cases: 0 of 72, and no void passed. Gemini 4 and Grok 4.7 refused to write attacks, and Claude Sonnet 5.5 produced none we could use; they are not counted.

What ran on a real rail, and what didn't

  • Real Braintree sandbox: the November attack run (126 runs per condition), the worst case, the live demo (checkout, refunds, voids, dispute card, webhooks) and the MCP server run.
  • In-memory rail with the sandbox's card limits: the 270 runs, so nine models √ó three repeats cost no sandbox noise. Real models decided every one of them.
  • Not run against PayPal REST: the Toolkit path (toolkit.py ‚Üí paypal.py) is tested with a faked HTTP layer only, because no PayPal account can be opened from T√ºrkiye.
  • Mocked PayPal inside Braintree does not enforce the refund total; no stop that only this gap explains is counted for Counterfoil.

Design

HOMI is a fictional homeware store. The site opens on the PayPal payment, then shows the refund request, the gate's decision and what reached Braintree. In the live demo a Blender model of the store replays each test step by step. Nothing plays on its own; every step is a click. The dashboard explains the system in plain words first and keeps the evidence one click away: the results band, a comparison with the alternatives, the review desk for held calls and every recorded agent call.

What's new here

Guards for agent payments sit on the buyer's side: an agent spends for a customer, and the guard limits how much and where (caps, allowlists, signed intent, approval over a threshold). A returns agent works on the seller's side, giving the store's money back, and the question there is different: is this money owed? A cap can't tell a USD 60 refund that is owed from one that isn't; it moved wrong money in 100 of 270 runs. Counterfoil answers with the store's return record and Braintree's own line items, and leaves to Braintree what Braintree already refuses.

Challenges we ran into

  • Braintree changed the design. Our first attack, a refund above the order total, is something Braintree already refuses on cards, so it couldn't be our selling point. We rebuilt the gate around the return record, which only the merchant has.
  • Mocked PayPal is too permissive. It accepted refunds above the sale total. We never count that as a Counterfoil catch.
  • Honest scoring. An injection the model ignores is not a gate success, and a blocked read is not a leak. Separating "agent misled" from "harm" is what makes the zero mean something.
  • Our own runs found bugs in the gate. An earlier gate let qwen3 push an inflated reorder through in all three repeats; that led to the rule that every charge is confirmed by a person. Each fix came with a test.

Accomplishments that we're proud of

  • Zero wrong money in 126 attack runs on the real Braintree sandbox and in 270 across nine models in simulation, while completing the same genuine requests as no gate (102 of 183) and handling 82% right overall.
  • Attacks that pass both baselines: over-refunds, split refunds and replays stay below the sale total, so neither Braintree nor a cap stops them.
  • A chargeback locks refunds before the agent asks, through Braintree's own webhook.
  • One config entry puts the gate in front of any MCP client that speaks the Toolkit's create_refund.

What we learned

  • Payment rails protect the transaction, not the business rule. The gap is the merchant's to close, with the merchant's own records, not with smarter prompts.
  • Better models are fooled less often, not never. With no gate, Claude Sonnet 5 was misled in 5 of 21 dev attack runs, qwen2.5 in 12, llama3.1 in 18.
  • Measure attempts and harm separately, or a model that ignores an injection looks like a gate success.

What's next

  • PayPal REST, measured: run the Toolkit path against a real PayPal REST sandbox.
  • More evidence: attacks written by people outside the project, more repeats on other days, more models.
  • Agent-initiated payments: the same record check in front of payments an agent starts, for example over open protocols such as x402. Not built or tested yet.
  • A store's own settings: refund window, automatic limit and who approves, editable by the merchant.
  • Privacy in a real store (KVKK / GDPR): tell customers an AI agent reads their e-mail and staff review held calls, which also answers the right to object to purely automated decisions.

What I learned

What's next for Counterfoil

Built With

Share this project:

Updates

Submission history