Inspiration
AI commerce agents can produce plausible payment actions even when user intent has drifted, been cancelled, replayed, or detached from the current proposal. We wanted a way to test that failure mode before real money is touched.
What it does
Agentic Commerce Stress Lab generates metamorphic and differential variants of a user mandate and conversation/handoff trace. Semantically irrelevant changes should preserve the payment decision; material changes such as amount drift, cancellation, replay, forged or detached confirmation must force BLOCK or CLARIFY. The lab runs multiple local LLM agents, compares their raw actions and authorization lineage, then sends only boundary-approved attempts to PayPal Sandbox.
On a controlled 60-trace corpus, Qwen2.5 and SmolLM2 both chose PAY on all 40 material attack traces. The deterministic boundary contained all 60/60 traces. We also verified authenticated PayPal Sandbox OAuth and an Orders v2 create-order: a blocked negative control made no provider call, while a known-safe ALLOW trace produced a Sandbox order with status CREATED.
How we built it
Python, local GGUF models through llama.cpp, deterministic authorization facts, intent lineage, HMAC-sealed intent, differential and metamorphic evaluators, a zero-dependency web dashboard, and PayPal Sandbox Orders v2. CI runs 27 tests and the repository includes sanitized evidence snapshots for clean-clone review.
Challenges we ran into
Distinguishing model behavior from adapter behavior; avoiding benchmark leakage; keeping model-generated tests separate from deterministic payment authorization; preserving reproducibility across small local models and CUDA/CPU backends; and proving that BLOCK paths make zero provider calls.
Accomplishments that we are proud of
Authenticated end-to-end PayPal Sandbox verification, cross-model differential evidence, a 240-trace lineage benchmark, an 80-case metamorphic suite, clean-clone dashboard evidence, public CI, and a PayPal client that refuses the live production host.
What we learned
LLM confidence is not authorization. A model can help generate adversarial tests and propose actions, but money movement should depend on compact, auditable invariants tied to current user consent.
What is next
Expand to more models, richer agent frameworks, multi-turn tool handoffs, fuzzing from real anonymized traces, and CI integrations that gate agent releases on payment-safety invariants.
Log in or sign up for Devpost to join the conversation.