CartHold

A payment-hold firewall for AI shopping agents. Gemini shops, CartHold checks every order, and PayPal holds the money until you release it.

Comment

Inspiration

AI agents are starting to buy things for people, and shoppers don't trust them with a wallet yet. In Nuvei's agentic-commerce survey, 61% of US shoppers said spending limits would make them comfortable with AI agents, 58-59% want human approval checkpoints (the page says 59% in one sentence and 58% in its executive summary), and only 1% want fully automated buying (Nuvei). The attack that worried us most is the boring one: a store page with a hidden sentence telling the shopping assistant to add something to the cart. The agent reads it, believes it, and pays. We wanted a layer between the agent and the money that does not depend on the agent behaving.

What it does

CartHold sits between a Gemini shopping agent and the PayPal Orders and Payments APIs. Before any money moves, it checks each order the agent proposes, and a second Gemini call reads the store pages the agent read, looking for hidden instructions the order follows. Then it gives one of three verdicts:

  • Allow: the order fits your mandate and stays under the auto-pay limit ($25 per demo run). CartHold authorizes and captures it through PayPal right away.
  • Hold: the order fits the mandate but goes past the auto-pay limit. You approve it as the buyer in PayPal, CartHold authorizes it so the funds are held, and then you press Release (capture) or Cancel (void). There are two 5-minute windows: if nobody approves in PayPal within 5 minutes, the order expires and nothing is held; once it is held, an authorization nobody releases within 5 minutes is voided.
  • Block: the order breaks the mandate, or it follows instructions hidden in a store page the agent read. No PayPal order is created, and the UI highlights the sentence the agent obeyed.

Every decision lands in an AG Grid decision log with the PayPal order, authorization and capture ids.

The demo opens in Simulated mode: a scripted compromised agent replays a known attack (a $30 mug plus a $19 coffee kit pushed by a hidden note) through the real guard, and the page inspector runs Gemini plus regex rules, as in Live mode. The scripted agent itself makes no Gemini calls. The $49 order is blocked, the agent retries with only the mug, and that $30 order goes through the PayPal sandbox for real. Live mode lets Gemini shop the demo stores on its own.

How we built it

  • PayPal: every order that passes the guard is created with Orders v2 intent=AUTHORIZE. Allow orders are paid by a sandbox test card. Hold orders use payment_source.paypal, so PayPal returns PAYER_ACTION_REQUIRED and a payer-action link; the buyer approves in PayPal, PayPal redirects back, and CartHold calls POST /v2/checkout/orders/{id}/authorize. Release calls Payments v2 capture and Cancel (or the timeout) calls void. A registered webhook gives us PayPal's own event log, which CartHold reads after a crash to void any authorization it left open.
  • Gemini: gemini-3.5-flash-lite runs the shopping agent (tool calls: read a store page, propose an order) and a second job as the LLM half of the page inspector.
  • The guard has two independent checks. A deterministic policy engine (web/src/lib/policy) enforces the server-owned mandate ($100 budget, $60 per item, max 2 units, three allowed stores) and provenance: every item and quantity must trace back to the words of the request. The inspector (web/src/lib/inspector) reads the pages the agent actually read, including hidden text, alt text and comments. A regex backstop always runs; Gemini can add a block only with a verbatim quote that the code verifies and that the order actually follows.
  • App: Next.js 16, React, TypeScript, Tailwind, Zod for validation, AG Grid Community for the log, Vitest for tests (fake PayPal and LLM clients, no keys needed). Hosted on Render's free tier.
  • Only the browser tab that started a run can approve, release or cancel it (a per-run owner token; the server keeps only its SHA-256 hash).

Challenges we ran into

  • We first wanted PayPal Vault v3, so the buyer saves PayPal once and the agent pays later without a redirect. On our sandbox app POST /v3/vault/setup-tokens returned 403 NOT_AUTHORIZED four times, even after enabling the vault toggles (debug ids in docs/evidence/2026-10-31-vault-403.md). We switched holds to per-order buyer approval, which is what ships.
  • Our regex inspector caught 19/20 attacks on the corpus we built it alongside, and only 4/12 on a held-out set we wrote later. That gap told us the corpus number was overfit, so we stopped quoting it as a detection rate. The LLM check already had a verified-quote rule, but it could only block on hidden or regex-flagged text; we extended it to visible text aimed at agents, with a code check that the order follows the quote. On a second, newer held-out set the full inspector caught 4/6 with 0/4 false positives; the regex alone caught 1/6. Small samples, and the set was written by us, by an author who had already read the inspector's visible-quote rule, so its full-inspector number is weaker evidence than its regex number (docs/evidence/2026-10-31-measurements.md).
  • Holds and their 5-minute timers live in memory, and Render's free tier sleeps and restarts. We void open holds on SIGTERM and recover from PayPal's webhook event log after a crash. We have not yet seen the SIGTERM path fire on the live host; the one hold we left open across a redeploy was voided by its own timer first.

Accomplishments that we're proud of

  • The full hold flow works on the hosted app with a real sandbox buyer. On 2026-10-31 a default Simulated run blocked the $49 order, a sandbox personal account approved the $30 mug in the PayPal popup, and Release captured it. Checked through the PayPal API: order 09G61688HE9076328 COMPLETED with payment_source paypal, capture 4854706228667701D COMPLETED for $30.00 (docs/evidence/2026-10-31-hosted-simulated.md).
  • The live Gemini agent really does get hijacked, and CartHold stops it. In live run ddc3af6a on the hosted app (2026-10-30, made with the gemini-flash-lite-latest alias before we pinned gemini-3.5-flash-lite on 2026-10-31), Gemini followed the hidden Northpine note and proposed the mug plus the coffee kit; CartHold blocked it, Gemini retried with the mug alone, and that order was held, approved and captured (order 0B575156XB8641420, capture 4GK99036D6729964W; docs/evidence/2026-10-29-live-runs.md).
  • In measured runs, the agent proposed the injected add-on in 10/10 runs and then 5/5 after fixes, and none of those orders got past CartHold (0/10, 0/5) (measurements).

What we learned

  • An agent can start from the right request and still pay for something nobody asked for. Checking the order against the request's own words blocked every hijacked Northpine order in our measured runs, without any model judgement.
  • An LLM is a useful second reader of a page, but we only let it block when code can confirm the quote it gives. Its unverified flags were often right, and the verification step still cost us some catches (a Spanish attack whose wording did not match the order line).
  • Measure on pages you did not tune against. Our first number looked great and was mostly overfitting.

What's next

  • Saved PayPal (Vault) once our sandbox account gets the scope, so holds don't need a redirect per order.
  • Persistent hold storage instead of in-memory timers.
  • A hidden-text extractor for arbitrary pages (today only the demo stores have one), and a larger held-out attack set written by someone other than us.
  • Mandates the user sets per merchant or category.

Built With

Share this project:

Updates

Submission history