PayProbe

Inspiration

An AI assistant will write you a PayPal checkout in an afternoon. What it tends to leave out is the boring part: checking that a webhook really came from PayPal, taking the price from the server instead of the browser, ignoring an event it has already handled, keeping the client secret out of the code.

None of these bugs throws an error. The checkout works, the demo works, and then someone pays $0.01 for a $100 item.

We review AI-built apps at BuildWithThis, and payment code is where a small mistake costs real money. Most scanners stop at a warning, and a founder who didn't write the code can't tell how serious a warning is. We wanted a tool that shows the bug happening.

What it does

PayProbe is a command-line tool you point at a project folder. It does three things.

Scan. It reads the code and flags PayPal-specific bugs, each with the file, the line, what an attacker gets out of it, and how to fix it:

• PP-WEBHOOK-UNVERIFIED: the webhook handler acts on events without checking PayPal's signature
• PP-AMOUNT-TAMPERING: the order amount comes from the client request
• PP-IDEMPOTENCY: no protection against the same event being handled twice
• PP-SECRET-LEAK and PP-SECRET-IN-CLIENT: a client secret in the source or in the browser bundle (PayProbe reports where, never the value)
• PP-ENV-MISMATCH: a hardcoded sandbox or live API URL
• PP-CLIENT-SIDE-ORDER: orders created or captured in the browser

Prove. For the two worst bugs it runs the attack against a URL you give it. It sends a forged PAYMENT.CAPTURE.COMPLETED webhook and checks whether the order gets fulfilled, then asks for a $100 item at $0.01 and checks which price the server uses. The repo includes a deliberately vulnerable demo shop and a fixed copy. On the vulnerable one both attacks work; on the fixed one both fail.

PayProbe also talks to the real PayPal sandbox. node src/cli.js prove --sandbox gets an OAuth token and creates an Orders v2 order on the server, with the amount set by the server, which is the safe way to do it.

Fix. With --ai, a language model ranks the findings by how much money is at risk and drafts a patch for each one, in plain words a non-security founder can follow. The patches are suggestions to review, not auto-applied.

On the bundled vulnerable shop, PayProbe finds 5 bugs (3 critical, 2 high). On the fixed shop it finds none.

Live sandbox proof

We ran prove --sandbox against a real PayPal sandbox app three times on Oct 30, 2026. Each run got a token and PayPal created a new order with status CREATED. The order ids were 6NP81211G2050282J, 68249394F2361861F and 71299204LT471664L (the last one is the run shown in the video). The output shows the order id and status only, never the credentials.

How we built it

It's Node.js (version 20 or newer) with no runtime dependencies, so you can run it without npm install.

The PayPal side lives in src/paypal/:

• OAuth 2.0 client credentials against api-m.sandbox.paypal.com
• Orders v2 create and read on the server, with a PayPal-Request-Id header so a retried request doesn't create a second order
• Webhook signature verification two ways: PayPal's own /v1/notifications/verify-webhook-signature endpoint, and an offline check that verifies the RSA-SHA256 signature over transmissionId|transmissionTime|webhookId|crc32(rawBody) and only accepts a certificate URL on a paypal.com host. The fixed demo shop uses this code to reject the forged webhook.

The scanner is six rule modules in src/scanner/rules/. The AI layer (src/ai/) talks to any OpenAI-compatible endpoint: a local model through llama.cpp or Ollama, or a hosted API with a key. We tested it with qwen2.5-1.5b-instruct running locally in llama.cpp, which costs nothing. Without a model, PayProbe falls back to built-in explanations, so the scan still works offline.

There are 16 tests (node --test): the scanner rules, the CRC-32 and certificate-host checks in the webhook verifier, both live attacks against both demo shops, and a guard on AI output. All 16 pass on a fresh clone; we ran them on Node 24.

How we used AI to build it. PayProbe was built with AI coding assistants. AI agents wrote most of the code, the tests and the docs.

Challenges we ran into

Webhook verification is easy to get subtly wrong. The CRC-32 has to be computed over the raw request bytes, not the JSON after you parse and re-serialize it, and the certificate URL has to be pinned to PayPal or an attacker can sign events with their own certificate. Both checks are written out in the repo with tests, because a "proof" built on a stub verifier would prove nothing.

The small local model caused two problems. When we asked it to explain the risk "in money terms", it invented loss figures, one of them $100,000,000,000. The prompt now forbids made-up numbers. It also sometimes got stuck repeating the same line of code until it hit the token limit. PayProbe now detects that and falls back to its built-in fix text instead of showing a broken patch.

We hit two bugs in our own setup too. The README told you to start the two demo shops and then run the tests, but the tests wanted the same ports and failed; they now ask the operating system for free ports. And node --test test/ fails on Node 22 and newer, which read test/ as a file, so the docs now say node --test.

Accomplishments that we're proud of

Every critical finding on the demo shop comes with an attack you can watch succeed, and then watch fail after the fix. That before-and-after is what we wanted a founder to see.

It runs with nothing installed except Node. The scan and attack demos need no accounts at all, and the sandbox demo needs only a free PayPal developer app.

What we learned

A warning is easy to ignore. "Your server accepted $0.01 for a $100 item" isn't.

A small free model is good enough to rank and explain findings. Its code patches are hit and miss, so we label them as suggestions.

What's next for PayProbe

These are plans, not built yet:

• Checks for PayPal subscriptions and Braintree
• A GitHub Action that runs the scan on every pull request
• Testing the AI layer with larger hosted models to compare patch quality

Built With

• javascript
• llama.cpp
• node.js
• openai-compatible-api
• paypal-orders-v2
• paypal-rest-api
• paypal-webhooks
• qwen2.5