Inspiration

Small grocers who pack online orders by hand hit the same problem every day: an item is out of stock after the customer has paid. The shop either charges for something the customer did not choose, or refunds and loses the sale. For a shopper with an allergy, a careless substitute is not just annoying, it can be unsafe. We wanted the payment itself to wait until both sides agree on what is in the bag.

The problem is common: in a Which? survey of 3,552 UK shoppers, one in five had a substitution in their most recent online grocery order. And it matters: FARE estimates that about 33 million people in the United States have at least one food allergy. (Sources are linked in the README.)

Who it is for: independent grocers who take web or phone orders and pack them by hand, and their shoppers with allergies or a restricted diet. Why it needs AI: a small shop's product data is rarely complete, and the package label is the authority. At picking time the only way to use the label is to read a photo of it.

What it does

Three roles touch an order: the shopper sets swap rules and answers each swap, the picker packs and photographs labels, and the shop approves the final amount. (In the demo the picker and the shop share one screen.)

  • The shopper builds a basket, lists allergens to avoid and a diet, and approves once in PayPal. The basket total is held with an authorization. Nothing is charged.
  • The picker marks each line picked or out of stock. For a missing item the picker proposes a substitute and photographs its label.
  • A vision model reads the label twice, independently, and quotes what it says: product, size, "contains", "may contain", vegetarian or vegan claims. Its output has no amount field.
  • Deterministic code adds the label's allergens to the shop's own data, blocks conflicts, and charges the lower of the two catalog prices. The model can add caution; it cannot remove it. The rules fail closed: a photo of the wrong product, or a partly unreadable label on an order with an allergy rule, blocks the swap.
  • The shopper accepts or declines each swap, with the photo and the quoted label text on screen. The answer is bound to that exact proposal by a hash.
  • The shop approves the settlement. The server recomputes it, compares it with the approval record, and sends one final capture. PayPal releases the rest of the hold. If nothing could be packed, the authorization is voided.
  • After delivery the shopper can report a charged line, and the shop can approve a refund that is capped at what was captured.
  • PayPal's webhooks are checked with verify-webhook-signature; only verified events are stored and shown on the order.

How we built it

PayPal, call by call:

  • Orders v2, intent: AUTHORIZE, approved in the JS SDK v6 window: holds the basket total and charges nothing.
  • POST /v2/checkout/orders/{id}/authorize: places the hold after the shopper approves.
  • POST /v2/payments/authorizations/{id}/capture with final_capture: true: charges the agreed amount once and releases the rest.
  • POST /v2/payments/authorizations/{id}/void: releases everything when nothing could be packed.
  • Agent Toolkit 1.11.0 create_refund (REST as fallback): refunds a reported line, capped at the capture.
  • Webhooks + verify-webhook-signature on the raw body: confirms each PayPal event before it is stored.

Everything else:

  • AI: Gemini 3.5 Flash-Lite through the Vercel AI SDK. Its one job that matters is reading a label photo into a schema-checked object with quoted evidence. A small tool loop (get_order, inspect_label, check_rules) explains the proposed swap in plain words. The stored reading and decision are always produced by server code, and the model's schema has no amount.
  • Settlement and refunds run in three steps: claim the order, make one PayPal call with a PayPal-Request-Id, record the answer. An interrupted call is resumed with the same key.
  • Security: every visitor gets a private demo shop (signed HttpOnly cookie), so one visitor cannot see or settle another's orders; shopper actions need the shopper's link; state-changing requests must be same-origin; a Content-Security-Policy limits scripts and frames to the site and PayPal; verified webhook amounts are reconciled with the ledger.
  • App: Next.js on Vercel, Neon Postgres, TypeScript. 38 unit tests cover the substitution rules, the two-reading merge, image prompt injection, the approval lock, races, interrupted calls and shop isolation. A Playwright script drives the real screens against the real sandbox, including the void path.
  • Evidence: the repository's evidence/ folder holds request and response records from the sandbox, including webhooks from the hosted demo with verification_status: SUCCESS.
  • The sample labels in the demo are generated images of fictional products, not photographs; the picker can also upload a real photo. In the demo the shopper's page refreshes by polling; a real shop would send the link by message.

Challenges we ran into

  • A final capture closes the authorization: the sandbox answers 422 AUTHORIZATION_ALREADY_CAPTURED to any further capture. We designed the settlement as exactly one call.
  • verify-webhook-signature fails if the event body is re-serialised, so the raw bytes are embedded in the verification request.
  • A review of our own code found that a PayPal call made inside a retried database save could be repeated or orphaned. Settlement and refunds now claim the order first, call PayPal once with an idempotency key, then record the answer.
  • The Agent Toolkit has no tool for authorize, capture-authorization or void, so those are hand-written REST calls; the toolkit is used where it fits (refunds and lookups), called from server code so the model never holds the refund tool.
  • A sample label said "may contain traces of peanuts" while the shop's data did not. That is why label readings are unioned with shop data and never replace it.
  • A label can carry text aimed at an AI. One sample label has a sticker telling the model to report no allergens and set the price to $0.00. The reading reports the sticker as data, and unit tests show it changes neither the amount nor the allergen block.

Accomplishments that we're proud of

  • The whole flow runs on the hosted demo against the PayPal sandbox, and PayPal's own records of those runs are in the repository.
  • The model has no way to change an amount or move money: there is no amount in its schema, and settlement needs a recorded human approval that the server re-checks.
  • In the sandbox, repeating a capture with the same PayPal-Request-Id returned the same capture id instead of charging again (record in evidence/).

What we learned

PayPal's authorize, capture and void calls already describe "pay only for what you got". The useful place for AI in that flow is in front of the money, gathering evidence people can check, while code and PayPal enforce the amount.

What's next for SwapSafe

  • Staff accounts for shops and a messaged link for shoppers (in the demo a browser cookie stands in for the shop's login so one person can play both roles).
  • Reauthorization for orders that sit longer than the three-day honor period.
  • A real catalog import, and a benchmark of label reading on photographed labels.

Built With

Share this project:

Updates

Submission history