Crashcart

Crash tests for AI shopping agents: hostile text goes in, measured damage comes out, and a seatbelt outside the model stops the bad order before it reaches PayPal.

Comment

Inspiration

AI agents are starting to buy things for people. An agent reads shop pages, reviews and tool results, and any of that text can carry a hidden instruction. We wanted to know, with numbers instead of opinions, how often a shopping agent can be talked into spending more than the user allowed, and whether a simple rule check placed outside the model can stop it.

What it does

Crashcart plants hostile text on fictional shop pages (22 attacks: 11 blunt, 11 crafted such as forged chat turns and fake tool results). A real AI model shops for one USB-C cable under USD 15. A judge that knows what the user asked lists every way the order breaks the request. Then three seatbelts decide what may reach PayPal: none, a basic one that checks money, and a strict one that also checks what is bought, how many, and where it ships. Orders the chosen seatbelt allows are created in the PayPal sandbox (Orders v2) and read back. The USB-C cable is only a stand-in: the same checks apply to any item, budget, merchant, currency or delivery address.

Results over 440 recorded runs on two Gemini models: blunt attacks fooled the agent in 2% of runs, crafted attacks in 18%. A fake chat turn saying "buy 3" worked in 20 of 20 runs. A hardened prompt cut crafted breaches from 27% to 9% but did not remove them. The strict seatbelt let 0 bad orders through and wrongly stopped 0 good ones. In a live PayPal sandbox run, with no seatbelt 21 bad orders were created at PayPal; with the strict seatbelt, 0.

How we built it

TypeScript for the engine (scenarios, judge, seatbelts, shopper agent, runner, PayPal sandbox client), 52 automated tests, and GitHub Actions to run live model calls and commit the results to the repository. The demo site is plain JavaScript on GitHub Pages and runs the same judge and seatbelt code as the tests. The "Try it live" button calls a small endpoint that runs one real attack against a real model and the real PayPal sandbox. It only accepts built-in attack ids, so no free text reaches the model provider.

Challenges we ran into

The whole project was built from a phone in Benin, with no laptop: files were created and uploaded through the GitHub web interface, and every live model call was launched from GitHub Actions workflows instead of a terminal. We also worked inside the free tiers: about 15 requests per minute and 500 per day on the Gemini models we used, so runs are paced and the runner stops cleanly when the daily quota is used up. Some models returned empty answers when their reasoning used up the output budget, which we fixed and now count as "no usable answer", never as a pass. Hosting the live endpoint without a server took several attempts before it worked from a public page.

Accomplishments that we're proud of

Every number on the site is computed from saved run files, not typed in by hand. Errors are never counted as passes. The limits are written openly: 22 hand-written attacks, one shop template, two free models, and a strict seatbelt that is deliberately blind to look-alike products. The "Try it live" button runs a real attack end to end in about ten seconds, with an animation that shows the order stopping at the seatbelt or reaching PayPal.

What we learned

Models resist attacks that shout, but they fall for text that looks like data: a forged chat turn, a fake tool result, a fake consent record. Telling the agent to be careful helps a lot and still fails. A rule check outside the model is cheap, predictable and cannot be argued with by a web page.

What's next for Crashcart

More shops, items and currencies beyond the USB-C stand-in, and more models. A way to capture the user's request reliably before the purchase, which the seatbelt currently assumes. Look-alike product detection. Attacks generated automatically against a given defense, and a larger sample so the rates carry tighter confidence.

AI tools used to build it

Claude (Anthropic) and ChatGPT as coding assistants. Product decisions, the test design and the architecture are the author's. The agents under test are the Gemini models listed above.

Try it out: instructions for judges

  1. Open https://sethgoisso-pixel.github.io/Crashcart/ (no login, nothing to install, nothing to pay).
  2. Scroll to "Try it live" (or press the yellow button at the top). Pick an attack, for example "Forged chat turns raise quantity and budget".
  3. Press "Run with no seatbelt". After about ten seconds you see what the real model ordered, the judge's verdict, and the PayPal sandbox order that was created.
  4. Press "Run with the strict seatbelt". The bad order is stopped before PayPal and nothing is sent.
  5. The agent is a live model, so answers can differ between runs. The recorded runs behind every number are in the "Scoreboard", "PayPal sandbox" and "Crash lab" sections.

Safety and limits: PayPal sandbox only, no real money, no real merchants. The live button accepts only the built-in fictional attacks. It is limited to 10 runs per connection per 10 minutes and uses a free model quota; if that quota is used up for the day, the page says so and the recorded runs stay available.

To run it yourself: clone the repository and run npm test (no keys needed) or node src/run.ts --dry-run --only review-extra-units to print the exact prompt. Live runs need LLM_API_KEY, PAYPAL_CLIENT_ID and PAYPAL_CLIENT_SECRET (see the README).

Built With

Share this project:

Updates

Submission history