Inspiration
Every agentic-commerce demo wires an AI agent straight to a wallet. Nobody ships the brakes. A human shops one item at a time; an agent can retry, parallelise, and spend at machine speed — at 3 a.m., on a new device, before anyone wakes up. Gatekeeper is the trust layer that has to exist before agentic commerce is deployable: one choke point between agent intent and payment execution.
What it does
An AI shopping agent proposes a payment. Gatekeeper scores it and returns one of three verdicts before PayPal is touched:
- ALLOW — the gateway creates and captures a PayPal sandbox order.
- REVIEW — the payment is paused in a human approval queue. No PayPal order exists yet; a reviewer approves or rejects it.
- BLOCK — stopped outright, with the factors that caused it logged.
Two scorers feed the same decision. A fast behavioural scorer drives the
live demo ($49 daytime → 5.7% ALLOW; $2,000 at 3 a.m. from a new device →
45.0% REVIEW; $12,000 3 a.m. burst → 78.8% BLOCK; thresholds 0.35 / 0.75).
The trained model scores real transaction rows at POST /propose-ieee.
PayPal sandbox Orders v2 handles create / approve / capture, and a
Disputes v1 client drafts chargeback evidence for merchant review — never
auto-submitted.
How we built it
- Risk model: LightGBM trained on the full IEEE-CIS Fraud Detection dataset (590,540 rows, 3.50% fraud, no subsampling), 70/15/15 stratified split, seed 42. 450 features: all V1–V339, identity left-joined, plus train-only feature engineering (transaction day/hour, frequency encoding, per-card mean amounts, V-missing count). Trained on Kaggle (private CPU kernel); model selection on calibration AP only, thresholds chosen on calibration only.
- Gateway: FastAPI (
gateway/app.py) with an execute/capture choke point — the agent can only pay through the gateway, after an ALLOW. - PayPal: Orders v2 client (OAuth token caching, create/get/capture) and Disputes v1 client, verified against the real PayPal sandbox: a real order reached CREATED with an approve link. Capture needs the payer's approval, per the Orders lifecycle.
- Demo: stdlib-only demo (
demo/run_demo.py, no installs, no credentials), a mock-PayPal end-to-end demo, an approval-queue UI, and an interactive live demo page.
Measured results (real IEEE-CIS, held-out test n=88,581, 3,099 fraud)
- ROC AUC 0.9771, average precision 0.8711
- REVIEW threshold 0.0455: precision 0.4437, recall 0.9051 — catches ~91% of fraud while reviewing only 7.1% of transactions
- BLOCK threshold 0.5750: precision 0.9287, recall 0.7477 — three quarters of fraud stopped outright
- Expected loss per 10,000 transactions: gated $6,587 vs no-gate $51,593 (review $2, false block $15, fraud loss = amount)
Challenges we ran into
- Prevalence distortion: our first training run kept all fraud rows and subsampled only legitimate ones, inflating the fraud rate to 13.8%. We caught it on review, discarded the run, fixed the sampler to preserve prevalence, and retrained.
- Review precision is the honest weakness: at 3.5% fraud prevalence, review precision is 44.4% — most reviewed transactions are legitimate. That is the price of catching 90%+ of fraud, and it is half the review burden of the earlier model (30.8% → 7.1%).
- A better-looking model that lost: the v5 ensemble scored slightly higher on AP (0.8724) but raised expected loss to $6,678, so it was not promoted. v4 stays live.
- The demo scorer is not the model: the browser demo's thresholds (0.35 / 0.75) are the behavioural scorer's, not the trained model's (0.0455 / 0.5750). The video and README say so explicitly.
Accomplishments that we're proud of
- A measured gate, not a vibe: precision-recall at the operating threshold and expected-loss math, instead of accuracy on a 3.5%-fraud dataset.
- Feature engineering beat model shopping: AP went 0.7785 (v3) → 0.8711 (v4) on the same data and split.
- A real sandbox order, verified CREATED — not just a mock.
- A demo video with a real screen-recorded live demo segment, and a
public repo with the 41 MB v4 model included so anyone can run
/propose-ieeeout of the box.
What we learned
- Environment and data discipline beat model tweaks: the biggest wins came from fixing sampling, calibrating on a held-out split, and choosing thresholds on calibration only.
- A promotion rule has to be written down before the run: AP up, AUC not down, gated loss not up — otherwise a slightly-higher-AP model that costs more would have shipped.
What's next for Gatekeeper
- Loss-optimal thresholds as a merchant setting (not a replacement for the calibrated operating point).
- Post-payment signals from Disputes v1 feeding back into scoring.
Built With
- detection
- disputes
- fastapi
- fraud
- ieee-cis
- kaggle
- lightgbm
- paypal
- python
- scikit-learn