Inspiration

Every agentic-commerce demo wires an AI agent straight to a wallet. Nobody ships the brakes. A human shops one item at a time; an agent can retry, parallelise, and spend at machine speed — at 3 a.m., on a new device, before anyone wakes up. Gatekeeper is the trust layer that has to exist before agentic commerce is deployable: one choke point between agent intent and payment execution.

What it does

An AI shopping agent proposes a payment. Gatekeeper scores it and returns one of three verdicts before PayPal is touched:

  • ALLOW — the gateway creates and captures a PayPal sandbox order.
  • REVIEW — the payment is paused in a human approval queue. No PayPal order exists yet; a reviewer approves or rejects it.
  • BLOCK — stopped outright, with the factors that caused it logged.

Two scorers feed the same decision. A fast behavioural scorer drives the live demo ($49 daytime → 5.7% ALLOW; $2,000 at 3 a.m. from a new device → 45.0% REVIEW; $12,000 3 a.m. burst → 78.8% BLOCK; thresholds 0.35 / 0.75). The trained model scores real transaction rows at POST /propose-ieee. PayPal sandbox Orders v2 handles create / approve / capture, and a Disputes v1 client drafts chargeback evidence for merchant review — never auto-submitted.

How we built it

  • Risk model: LightGBM trained on the full IEEE-CIS Fraud Detection dataset (590,540 rows, 3.50% fraud, no subsampling), 70/15/15 stratified split, seed 42. 450 features: all V1–V339, identity left-joined, plus train-only feature engineering (transaction day/hour, frequency encoding, per-card mean amounts, V-missing count). Trained on Kaggle (private CPU kernel); model selection on calibration AP only, thresholds chosen on calibration only.
  • Gateway: FastAPI (gateway/app.py) with an execute/capture choke point — the agent can only pay through the gateway, after an ALLOW.
  • PayPal: Orders v2 client (OAuth token caching, create/get/capture) and Disputes v1 client, verified against the real PayPal sandbox: a real order reached CREATED with an approve link. Capture needs the payer's approval, per the Orders lifecycle.
  • Demo: stdlib-only demo (demo/run_demo.py, no installs, no credentials), a mock-PayPal end-to-end demo, an approval-queue UI, and an interactive live demo page.

Measured results (real IEEE-CIS, held-out test n=88,581, 3,099 fraud)

  • ROC AUC 0.9771, average precision 0.8711
  • REVIEW threshold 0.0455: precision 0.4437, recall 0.9051 — catches ~91% of fraud while reviewing only 7.1% of transactions
  • BLOCK threshold 0.5750: precision 0.9287, recall 0.7477 — three quarters of fraud stopped outright
  • Expected loss per 10,000 transactions: gated $6,587 vs no-gate $51,593 (review $2, false block $15, fraud loss = amount)

Challenges we ran into

  • Prevalence distortion: our first training run kept all fraud rows and subsampled only legitimate ones, inflating the fraud rate to 13.8%. We caught it on review, discarded the run, fixed the sampler to preserve prevalence, and retrained.
  • Review precision is the honest weakness: at 3.5% fraud prevalence, review precision is 44.4% — most reviewed transactions are legitimate. That is the price of catching 90%+ of fraud, and it is half the review burden of the earlier model (30.8% → 7.1%).
  • A better-looking model that lost: the v5 ensemble scored slightly higher on AP (0.8724) but raised expected loss to $6,678, so it was not promoted. v4 stays live.
  • The demo scorer is not the model: the browser demo's thresholds (0.35 / 0.75) are the behavioural scorer's, not the trained model's (0.0455 / 0.5750). The video and README say so explicitly.

Accomplishments that we're proud of

  • A measured gate, not a vibe: precision-recall at the operating threshold and expected-loss math, instead of accuracy on a 3.5%-fraud dataset.
  • Feature engineering beat model shopping: AP went 0.7785 (v3) → 0.8711 (v4) on the same data and split.
  • A real sandbox order, verified CREATED — not just a mock.
  • A demo video with a real screen-recorded live demo segment, and a public repo with the 41 MB v4 model included so anyone can run /propose-ieee out of the box.

What we learned

  • Environment and data discipline beat model tweaks: the biggest wins came from fixing sampling, calibrating on a held-out split, and choosing thresholds on calibration only.
  • A promotion rule has to be written down before the run: AP up, AUC not down, gated loss not up — otherwise a slightly-higher-AP model that costs more would have shipped.

What's next for Gatekeeper

  • Loss-optimal thresholds as a merchant setting (not a replacement for the calibrated operating point).
  • Post-payment signals from Disputes v1 feeding back into scoring.

Built With

Share this project:

Updates

Submission history