Checkout Sentinel

AI proposes defences for PayPal checkout pages against skimming attacks, and a verifier proves each fix blocks the attack while PayPal checkout still works, before a human approves it.

Comment

Inspiration

Payment pages load dozens of third-party scripts. One compromised vendor script can skim every field a buyer types, or draw a fake PayPal login over the real checkout. Since January 2025, merchants using PayPal buttons (PCI DSS SAQ A) must confirm their site is not susceptible to attacks from scripts, and requirements 6.4.3 and 11.6.1 apply to everyone else. Most small merchants have no way to produce that evidence, and blindly blocking scripts breaks checkout.

What it does

Checkout Sentinel watches a PayPal merchant's checkout page for skimming attacks.

  • Detect: a headless browser visits the page, records every script with its SHA-256 hash, types unique canary values into the form and goes through a real PayPal sandbox checkout. Deterministic rules flag injected or tampered scripts, listeners on sensitive fields from unauthorised code, data sent to unknown servers, fake payment overlays and weakened security headers.
  • Investigate: an AI analyst with read-only tools explains what the suspicious code does and where it sends data, classifies it (skimmer or benign vendor update), and proposes a remediation: a Content-Security-Policy, Subresource Integrity pins and inventory decisions. The proposal is data, never executed as code.
  • Prove: a verifier with no AI attacks the page again with the proposed policy enforced. The fix passes only if the canary never reaches the attacker, no overlay or rogue listener survives, the clean control page is unchanged and the PayPal sandbox checkout still reaches approval. Anything missing fails the verdict.
  • Approve: only a human click deploys the policy to the shop, followed by a post-deploy check that rolls back if the shop does not serve it. The dashboard produces PCI DSS v4.0.1-aligned evidence (script inventory with justification, change-detection log, header monitoring). It is evidence, not a compliance certification.

How we built it

Python 3.12 (typed, ruff and mypy clean), FastAPI, Playwright with raw CDP (Chromium), PostgreSQL, Docker Compose, NVIDIA Nemotron on Nebius through an OpenAI-compatible endpoint, htmx and AG Grid Community for the dashboard, pytest (unit, offline replay and browser tests against the real demo stack). The hosted demo runs on AWS (EC2 behind Caddy with TLS and basic auth and RDS PostgreSQL; a Lambda + SQS path for CSP violation reports is written and tested with fakes). CI runs Semgrep, pip-audit, gitleaks and an OWASP ZAP baseline scan.

Challenges we ran into

  • Making verification honest: a blocked attack is not automatically a successful fix, so the verifier also requires the PayPal sandbox checkout to still work and the clean control page to be unchanged.
  • PayPal's risk system challenges automated buyer logins, so a completed capture is not reliably automatable. Checkout evidence is "verified up to PayPal approval", and we say so everywhere instead of overclaiming.
  • Keeping the AI boxed in: read-only tools, proposals as data, a deterministic validator and a no-AI verifier between the model and the shop.
  • Making it judgeable with zero keys: the app runs offline from committed recordings and recorded AI responses.

Accomplishments that we're proud of

  • Detection on the four demo scenarios (clean control, benign vendor update, compromised vendor keylogger, fake PayPal overlay): precision 1.0, recall 1.0, zero false positives, including zero findings on the clean control.
  • AI investigation (nvidia/nemotron-3-super-120b-a12b): 9 of 9 recorded runs verified and correctly classified, on the first attempt, for $0.13 in total.
  • A working hosted demo and a local demo that runs end to end with no PayPal, LLM or AWS credentials.

What we learned

The hard part of AI security tooling is not getting the model to suggest a fix, it is proving the fix works without breaking payments. Putting a deterministic, fail-closed verifier between the model and production made the AI output trustworthy enough to approve.

What's next for Checkout Sentinel

  • CSP violation reports from real shoppers' browsers alongside the synthetic crawls (each crawl is one visit, a sample, not full coverage).
  • More attack scenarios and vendor-update patterns.
  • Packaging it so a merchant can point it at their own checkout page.

Built With

Share this project:

Updates

Submission history