Inspiration

I kept seeing the same PayPal project over and over: another shopping agent, another payment assistant.

Nobody was testing the checkout itself.

Then while I was playing with the PayPal sandbox, I did something stupid by accident - I sent the same payment twice. No request id, no lock, nothing. PayPal charged me twice. Two orders. Two captures. For one cart.

That scared me a little. If I could do that by accident, what could someone do on purpose?

So I stopped building another agent and built the opposite: a tool that tries to break checkouts on purpose, then checks PayPal's own records to see if it worked.

That's Crashpay.

What it does

Crashpay attacks a checkout in the PayPal sandbox, then checks what actually happened.

It doesn't trust what the shop says. It goes to PayPal and asks: GET /v2/checkout/orders/{id} - did you actually capture this money?

16 attack classes. 77 requests per shop. 96 scored cells per run.

Things like: double submit, burst of 8 at the same second, retry after timeout, selling a $300 monitor for $0.01, reusing a cart for a different item, sending 35 garbage bodies, sending a 2 MB body.

Then the loop that makes it different:

Attack -> AI writes a patch -> I read it -> I run it -> Same attacks again.

In one recorded run, the vulnerable shop went from 2 of 16 defended to 15 of 16 in two rounds. The one thing it never fixed was the 2 MB body. That failure is still in the README, because it's true.

Demo: https://youtu.be/wywT1qXjrtU

How we built it

Six shops, all mine, all on localhost. All take the same request: POST /pay with item, price, cart_id.

  • Vulnerable (5001): trusts the price from the browser, sends no request id, no validation, no size limit.
  • Hand-fixed (5002): price from its own list, per-cart locks, cart id sent as PayPal request id, only 1-64 letters/digits/dash/underscore allowed, refuses bodies over 100 KB.
  • AI v1 to v4 (5004-5007): written by gpt-oss-120b through Groq.

The scoreboard is a simple web page at 127.0.0.1:5000. You click Run all attacks, it runs, and you get a table: P or FAIL for every shop vs every attack.

The scoring rule I learned the hard way: count captures, not orders. A declined card still returns an order marked COMPLETED. Inside it, the capture says DECLINED. If you trust the order status, you think you got paid when you didn't.

The autofix loop in crashpay/autofix.py does the whole thing in one command. It takes the failing attack's own output, adds a one-line rule per class, asks the model for a patch, checks the imports, shows it to me for approval, starts it on a free port, and re-attacks it. No failure descriptions typed by me in that loop - the attacks write the prompt.

I used Claude as a coding assistant during development. I ran every command myself, read every patch before running it, and checked every result against PayPal's records.

Challenges we ran into

1. My first PayPal accounts were useless. I created them for India. PayPal's checkout page just failed. No clear error. It took me a while to realize I needed to recreate them with country set to United States. That one setting cost me half a day.

2. My "fixed" shop was not fixed. Crashpay found a bug in my own hand-fixed shop six times. Six.

The worst one: I thought PayPal-Request-Id alone was enough. It wasn't. Under simultaneous duplicates, a second charge got through in some runs - 7 of 10 defended once, 4 of 5 another time. I still don't know exactly why the header wasn't enough. I fixed it by adding per-cart locks and returning the first reply on a repeat.

Then it crashed on a missing cart id. Then on a list as an item. Then on a cart id that was just spaces. Then on newline, emoji, and right-to-left characters - because the requests library refuses those in a header. Then it accepted a 2 MB body.

Every time I thought "okay, now it's fixed," Crashpay found another one.

3. The AI patches lied by looking good. v1 passed everything I had at the time, then failed type confusion and 13 fuzz bodies. v2 and v3 kept accepting bad cart ids - empty string, true, 0, spaces. v4 finally passes fuzz, but still fails replay, restart, and oversized body. None of them send a request id to PayPal. None of them remember carts after a restart.

4. Flaky results. One run is not truth. The vulnerable shop scored 2, 3, and 4 of 16 in three different runs. Same code. I kept all three numbers in the README instead of picking the best one. The Nov 12 re-run had errors I never found the cause of. I left that in too.

5. The video. My laptop refused to record properly. I ended up doing it in short takes. The final video is 2:48. It's not pretty, but it shows the real loop working.

Accomplishments that we're proud of

I'm proud of three things, and none of them are "I used AI."

First, I proved a real PayPal behavior with PayPal's own records: without a request id, duplicate sends = duplicate charges. With the same request id, you get the same order back. That's not a theory. That's in the sandbox logs.

Second, I proved my own fix was broken six times, and fixed it six times. Most people would have stopped after the first "16/16" and called it done. Crashpay didn't let me.

Third, I proved where AI still fails. The loop went 2 -> 11 -> 15, then stopped. It never fixed the 2 MB body in any run. A one-line size limit fixed it in my hand-fixed shop. That tells you something important: a patch is only as good as the attacks that grade it.

What we learned

  • Count captures, not orders. Order status lies.
  • A request id is not a lock. You need both.
  • The attacks find the bugs. The model just fixes what you describe. In rounds v2 to v4, I told the model what failed. It didn't discover anything on its own.
  • Small samples lie. One run is a story, not proof. I need 20 runs, not 1.
  • Honest limits win trust. My hand-fixed shop's 16/16 is not independent evidence - I repaired it against these same attacks. Saying that out loud makes the rest of the project more believable, not less.

What's next for Crashpay

  1. Webhooks. Right now I only test the checkout request. Next is forged and replayed webhooks - can a shop tell a real PayPal notification from a fake one?
  2. Any checkout. Right now Crashpay only talks to my six shops. I want a config file: URL, field names, order id location. Point it at any checkout.
  3. A real database. Right now shops keep carts in memory. Restart them and they forget everything. That's why v4 charges twice after a restart. Give them a DB, add the restart check to the web page, and run another patch round.
  4. More runs. Repeat every attack 20 times, not once.

Crashpay is sandbox only. No real money moves. It's an independent project, not affiliated with PayPal.

Built With

Share this project:

Updates

Submission history