-
-
War Room during a simulated card-testing attack: scorecard says 99% card testing; 3-D Secure on, bot devices blocked, held payments voided
-
Incident report: what fired (detect), how the scorecard judged it, and which fixed safety rule allowed each PayPal action
-
On the owner's phone: installable PWA, push alerts via Firebase, Ask Guard on Gemini, and nothing changes until she taps Confirm
-
Demo shop Wick & Wax: real PayPal sandbox checkout with the PayPal button and PayPal-hosted card fields (JavaScript SDK)
-
Investigate page on AG Studio by AG Grid: live report, custom AI verdict widget, and the Guard analyst agent running on Gemini
-
Dispute assistant: rules recommend, Gemini reads the buyer's message and drafts the reply, the owner decides (PayPal Disputes API)
-
Connect store: API keys, copy-paste guard-check code for Node, Python, PHP and curl, plus SDKs generated with APIMatic
-
Demo lab: labelled simulated scenarios run through the real pipeline: card testing, stolen accounts, slow drip, stealth bot, flash sale
-
Product page: one night at a small PayPal shop, from attack detected to morning report
Inspiration
Criminals buy stolen card numbers and PayPal logins in bulk. Before they use them, they point a bot at a small shop's checkout and make hundreds of tiny purchases to find out which ones still work. This is card testing, and the shop owner pays for it: chargebacks and dispute fees, a PayPal risk review that can limit the account, and a morning spent refunding orders one by one.
Owners keep reporting it. A Hacker News thread titled "PayPal does not seem equipped to deal with this" got 291 points. Issues like "Bot farms testing multiple card numbers on my site" keep appearing on WooCommerce's PayPal plugin, which runs on 800,000+ stores, and its support forum had 23 attack-related threads between February and October 2026. Small charities describe bursts of fake "donations". Small shops have no fraud team, so nobody is awake when it happens.
Meet Meera (a made-up shop owner whose night matches the real reports). She sells hand-poured candles through PayPal. At 2 AM a bot starts buying her $3 tea-light pack with a different stolen card each time. By 7:30 AM she has 80 "orders", and over the following days she gets disputes and fees on all of them. Checkout Guard is the night guard she doesn't have.
What it does
Checkout Guard connects to a shop's PayPal checkout and works in four steps: Watch → Judge → Act → Explain.
- Watch. Before PayPal creates an order, the shop asks the guard about the shopper. PayPal webhooks report every capture, decline, refund and dispute. Statistical triggers (Poisson, binomial and CUSUM tests) compare tonight's traffic with the shop's own normal baseline.
- Judge. A transparent scorecard decides in milliseconds whether this is card testing, stolen-account testing, a genuine rush, a PayPal outage, or unclear. Each verdict comes with probability bars and reasons backed by our own numbers, such as "×12 vs normal". Google Gemini explains the verdict in plain English, and gives a second opinion only when the scorecard is unsure.
- Act. A fixed policy engine (rules R0–R5) decides what is allowed, and PayPal carries it out:
- 3-D Secure is switched on through the Orders API, only for first-time shoppers, so returning customers aren't slowed down.
- Attacking devices are blocked before PayPal creates an order. A network-level check catches bots that rotate their device id.
- The targeted item is held for first-time shoppers.
- Payment holds. A first-time shopper's payment is authorised, not captured, for a short review window. When an attack is confirmed, its held payments are voided: never charged, no PayPal fee, nothing left to dispute.
- Capped refunds cover any attack payment that settled before anything looked wrong. Caps are per payment, per incident and per day, and anything above a cap waits for the owner.
- Explain. The owner gets one plain-English push notification on their phone, a morning report, and an evidence pack ready to send to PayPal support.
It doesn't punish real customers. A flash sale with mostly returning shoppers is called a genuine rush: no blocks, no refunds, no friction. "I'm running a sale" mode tells the guard in advance. Once the attack goes quiet, the defences stand down by themselves.
Around that core:
- Ask Guard, an AI assistant on every page. Ask "Is anything waiting for me?" or "Pause the guard". It looks things up with tools, and anything that would change something comes back as a card with Confirm / Cancel. Only the owner's Confirm does anything.
- Investigate, built on AG Studio by AG Grid. Every attempt, incident and action from the last 48 hours forms an editable report, with our own AI verdict and Defence timeline widgets in a night theme. The Guard analyst agent, built on the AG Studio Agent Framework and running on Gemini, reads the incident and hands the dashboard work to Studio's own agents. Auto-investigate builds a page for each new incident.
- Dispute assistant. When a chargeback still lands, fixed rules recommend contesting or accepting from our records. Gemini reads the buyer's own message, flags when it contradicts the reason they picked, and drafts the reply. The owner decides, and only then do we call PayPal's Disputes API.
- Connect store, with three ways in:
- Observer mode works from PayPal webhooks alone, with no checkout change.
- One guard API call per checkout unlocks every defence.
- SDKs for TypeScript, Python and PHP are generated with APIMatic from our OpenAPI spec.
The page also has API keys, copy-paste snippets and a "Send test check" dry run.
- Installable PWA with push notifications through Firebase Cloud Messaging, so the alert finds the owner on their phone.
- Owner controls: refund caps, autopilot, a kill switch, the stand-down time and sale windows. Owner and viewer roles are checked on every request.
How PayPal is used (all in the PayPal sandbox)
PayPal is both the checkout being attacked and the hands that defend it.
- PayPal JavaScript SDK (v6): the shop's PayPal button and PayPal-hosted Card Fields. Card numbers go from the browser to PayPal and never reach our server. The bank's 3-D Secure window runs inside PayPal's fields.
- OAuth 2.0 + client tokens: server access token, and a browser-safe client token for the SDK.
- Orders API: create, capture and look up every checkout, with
PayPal-Request-Ididempotency and acustom_idthat links webhooks to attempts. Card orders sendSCA_ALWAYSwhile the guard is in 3-D Secure mode. Before capture we read the card'sauthentication_result(liability shift, enrollment, status) on our server, so a failed 3-D Secure challenge is never charged. - Orders API with
intent: AUTHORIZE+ Payments API authorizations: payment holds for first-time shoppers, then void (the attack's payments) or capture (everyone else). - Payments API refunds: capped refunds of settled attack captures, each with a stable request id, so a retry after a crash is sent exactly once.
- Disputes API:
accept-claimandprovide-evidence(multipart), only after the owner decides. - Webhooks: every event is verified with
verify-webhook-signatureagainst the raw body. Payer PII is redacted before storage, and unverified events are never acted on. - FraudNet correlation: the
PayPal-Client-Metadata-Idheader goes with each order, so PayPal's own risk engine can score it. We feed PayPal's tools rather than replace them. - Backend PayPal calls use PayPal's official Python
paypal-server-sdkplushttpx.
How AI is used
Deterministic code decides, and the AI explains. No AI output can move money, block a shopper or change a number the owner sees.
- Gemini explainer writes the owner-facing summary of each verdict. If its text argues for another class, or cites a number that doesn't match ours, it is replaced by our template.
- Gemini tie-breaker is asked only when the scorecard is unsure, and isn't shown the scorecard's answer. If it agrees, confidence is capped at 0.80, below the 0.85 refund floor. If it disagrees, the incident stays "unclear" and only soft defences apply.
- Gemini report writer words the phone alert, the morning report and the PayPal evidence pack. It may only use
{{placeholders}}, which we fill from the database. A draft that writes its own digits is thrown away. - Ask Guard is a Gemini chat with function calling. Look-up tools run on the server for the member's own shop. Change tools only return a proposal, checked against the database, and the owner must Confirm it.
- Guard analyst (AG Studio Agent Framework + Gemini) reads the guard's state through read-only tools and has Studio's built-in agents build the investigation dashboard. Each LLM turn goes through our backend, so the Gemini key stays on the server.
- Dispute assistant: Gemini reads the disputing buyer's messages against our records and drafts the note to PayPal. Every amount, date and id is filled in from our records.
- Models:
gemini-3.5-flash, falling back togemini-2.5-flash, thengemini-flash-lite-latest, through thegoogle-genaiSDK. - Guardrails:
- An evidence check drops claims that don't match our computed features (5 % tolerance).
- A number guard catches invented numbers in the prose.
- The AI sees only aggregates and masked rows, never card numbers, names or emails.
- A per-model circuit breaker is shared through Postgres.
- When Gemini is down or out of quota, the scorecard still decides and acts, and the text falls back to our template.
How we built it
- Backend: Python 3.12, FastAPI, SQLAlchemy + asyncpg + Alembic on PostgreSQL, and a worker for detection, holds, refunds and stand-down. Live updates go over server-sent events through Postgres LISTEN/NOTIFY. The policy engine is pure and unit-tested.
- Frontend: Next.js 16, React, TypeScript and Tailwind. AG Grid + AG Charts across the War Room, and AG Studio on the Investigate page. It's an installable PWA with a service worker and Firebase Cloud Messaging. Cloudflare Turnstile is the human check.
- Hosting: live on Render (one Docker container with the UI, API and worker), with Supabase Postgres. The repo also has a
render.yamlBlueprint for the full multi-service layout. GitHub Actions runs CI (ruff, migrations with a drift check, pytest against Postgres, tsc, lint, build), a daily health check, and a keep-alive. - APIMatic generated our TypeScript, Python and PHP SDKs from the public guard API's OpenAPI spec.
- Labelled attack simulator: PayPal's sandbox can't stage a real attack, so we built one. It runs card testing, stolen-account testing, slow drip, stealth bot, attack-during-a-sale, flash sale, busy evening, PayPal outage and normal night. Offline mode synthesises outcomes. Live mode creates real sandbox orders with sandbox test cards, and declines come from PayPal's
CCREJECT-*triggers. Every simulated row carries a SIM badge.
Evaluation. All of this is simulated. The weights were tuned on the same simulator, so it shows consistency, not real-world accuracy. We ran 13 scenarios × 20 seeds through the real triggers and scorecard:
- Card testing: detected 100 % of the time, median 10 s, final verdict correct 100 %.
- Stolen-account testing: correct 100 %. Slow drip and stealth bot: detected 100 %.
- Flash sales, announced or unannounced: 0 blocked or refunded.
- Normal nights: no trigger at all.
Challenges we ran into
- The sandbox can't stage an attack. We built the labelled simulator with offline and live modes, and drove real declines with PayPal's
CCREJECT-*name triggers. - LLMs are slow, rate-limited and non-deterministic. A Gemini call takes 3–25 s and free quota runs out. So the scorecard decides in milliseconds and the defences are committed before Gemini is asked for words. Circuit breakers and fallback models keep the explanations coming.
- Not punishing real customers was harder than catching bots. It took targeted 3-D Secure for first-time devices only, returning-customer detection, sale windows, and a stand-down after the attack.
- Refunds still cost the fixed fee. Authorise-then-void for first-time shoppers means the attack's payments never settle at all.
- A shared hosted demo on free hosting: we ran everything in one container, and added an idle janitor that restores defaults, run cooldowns and "changed by another visitor" notices.
- Real-world bugs, such as
window.paypalcolliding with browser extensions (Honey). We fixed that with our own SDK namespace.
Accomplishments that we're proud of
- The flow works end to end against the real PayPal sandbox:
- a 3-D Secure pass is captured and a 3-D Secure fail is never charged, through Card Fields;
- a held payment is authorised, then captured or voided;
- webhooks are verified;
- a sandbox dispute was answered through the Disputes API.
- From the first trigger to the first defence takes under a second, and a card-testing burst is detected in about 10 seconds.
- The AI is genuinely useful and can't do harm: every number the owner reads comes from our database, and every change needs a human Confirm.
- A shop can connect with no code (webhooks only), or with one API call and a generated SDK.
What we learned
- Card testing is an incident problem, not a single-payment problem. PayPal scores one payment at a time; Checkout Guard runs the response to the whole attack.
- The best use of an LLM in payments is judgment, language and investigation, sitting next to deterministic rules, not in place of them.
- Authorise-then-void is the cleanest defence: the money, the fee and the dispute never happen.
What's next for Checkout Guard
- A WooCommerce / PayPal plugin that calls the guard API, where most of the real reports come from.
- Baselines from real PayPal history (Transaction Search) instead of seeded history.
- Multi-merchant onboarding through PayPal's partner platform.
- Device and payer graph clustering, and a trained model once real labelled attack data exists.
- FraudNet's device-data script with a PayPal-issued source id.
What's real and what's simulated
Real:
- PayPal sandbox OAuth, orders, captures, authorisations, voids, refunds, 3-D Secure, Card Fields, disputes and webhook verification
- Gemini calls
- Postgres, the worker and the live dashboard
- Push notifications
Simulated, and labelled on screen:
- The attacker and the "stolen" credentials (sandbox test cards and synthetic buyers)
- 28 days of seeded "normal" history
- The PayPal outage scenario
The project was built from scratch for this hackathon (first commit 11 Oct 2026).
Try it
- Live demo: https://checkout-guard-8s7p.onrender.com. It runs on free hosting, so if it has been idle, the first page can take about a minute.
- Owner dashboard: /guard. Sign in as
[email protected]with the judge password from the testing instructions, or create your own account (it joins as a viewer). - Click path:
- Simulation bar → Card-testing attack, then watch the War Room.
- Open the incident.
- Run Flash sale (nothing should be blocked).
- Open Investigate (AG Studio).
- Open Ask Guard.
- Open Connect store.
- Code, setup and full docs: https://github.com/BrawlerXull/pp-hack (MIT)
Built With
- ag-charts
- ag-grid
- ag-studio
- apimatic
- cloudflare-turnstile
- docker
- fastapi
- firebase-cloud-messaging
- gemini
- google-genai
- next.js
- paypal
- paypal-disputes-api
- paypal-javascript-sdk
- paypal-orders-api
- paypal-payments-api
- paypal-webhooks
- postgresql
- python
- react
- render
- sqlalchemy
- supabase
- tailwindcss
- typescript