ASA - Agent Surface Audit

Black-box audit of PayPal's agent surface (@paypal/mcp + invoice REST, sandbox-only): every probe leaves a receipt; a deterministic checker, not the LLM, decides PASS / DRIFT / OBSERVE.

Comment

Inspiration

Merchants are handing agents refund and payment powers, and the existing answers are runtime guards or generic verifiers. Nobody had published a measured behavior model of PayPal's own agent tools on the sandbox: what a duplicate call actually does, which illegal states are actually accepted, what failure semantics an agent actually sees. The hook was one dumb-sounding question: "what does PayPal's official MCP toolkit do when you call it wrong?" The answer — create_invoice silently accepts a mis-keyed line_items field and stores a $20.00 invoice as a $0.00 object with HTTP 201 and no error anywhere — turned into asa, a receipt-first audit.

What it does

  • Probes 4 abuse classes (duplicate retries, illegal states, cumulative over-limit, untrusted echoes) across REST and MCP: 108 candidate cells, 96 excluded with a receipted reason each, 51 REST + 12 MCP executed (version-pinned @paypal/[email protected]).
  • Labels every cell PASS / DRIFT / OBSERVE and never claims more than its JSONL receipts (ids, debug_id, per-call flag of forced vs organic behavior).
  • Compiles merchant policies: LLM-drafted v0 (doc-faithful) FAILs against receipts; v1 is declared policy_status: known-hazard documentation and the deterministic checker re-runs it — the fail → document → recheck cycle is the demo, no repair-theater.
  • Measures the AI story: the exact masked-failure MCP text + the real create_invoice schema into agent probes — 0 hazard outcomes in 75 trials (NVIDIA n=20/cell + GLM n=5/cell, Wilson 95% upper bounds ≤ 0.43). The exposure NOT demonstrated (a client forwarding only the structured flag) stays a labeled hypothesis.
  • Replays the findings live — scripts/replay_findings.py, 7 bounded sandbox calls, 4/4 REPLAYED — and renders ONE offline report (report/index.html) built only from receipt-derived artifacts.

What existed before

Nothing — all code, artifacts, and docs in the repo were created during this submission period (N/A pre-existing). The "pre-existing" things are the objects under test: PayPal's MCP server and sandbox REST APIs.

How we built it

  • asa/ Python package: 108-cell matrix, JSONL receipt recorder (with id extraction), REST client + MCP stdio client, and a hard sandbox gate (config.py refuses non-sandbox credentials by construction).
  • Deterministic checker (asa/policy.py): compiled policies vs. receipted canon — the checker, never the LLM, decides verdicts; raw compile/draft transcripts committed in docs/llm_transcripts.md.
  • Agent probes: real masked tool text + real schema streamed to live models; per-cell counts scored by code with Wilson intervals in atlas/agent_probe.json.
  • The demo video itself is deterministic: a Pillow→ffmpeg renderer paints the committed JSON artifacts frame-by-frame, so every number on screen IS a receipt (Python + FFmpeg).

Challenges we faced

  • Success lies: the MCP tool returns isError=false while its own text says {"ok":false,"status":406} — we had to resist asserting and instead measure what agents do with that text, then add a schema-boundary check (the -32602 Required at items rejection also rides isError=false), and mark the unmeasured exposure path as a hypothesis.
  • 96 exclusions proved heavier than 63 runs — every excluded matrix cell needed its own receipted reason.
  • Honest v1: "fix the policy" was easy to fake; instead v1 carries a formal policy_status declaring it documents the hazard + merchant-side mitigation, no repair claim.
  • One candidate probe model family (Qwen) was unavailable at probe time through our access — recorded as UNAVAILABLE in the artifact, never substituted.
  • Zero-hazard results must not be over-read: 75 trials bounds hazard probability per cell (Wilson ≤ 0.16 at n=20), and we say exactly that.

What we learned

  • The most agent-dangerous failures look like receipts: silent 201s, $0 objects, masked failures — not 500s.
  • Idempotency contracts can be family-specific: Orders v2 honors PayPal-Request-Id; invoicing ignores it, so same-header retries duplicate money objects.
  • Bounded measurement beats narrative: 75 trials with intervals tell a judge more than a hundred "AI might..." lines.
  • Making the checker own every verdict is what made AI usage load-bearing and auditable.

Built with

Python 3.13, @paypal/mcp (stdio, pinned 1.8.1), PayPal Sandbox REST (Invoicing v2), MCP (Model Context Protocol), NVIDIA Nemotron 3 Ultra via NVIDIA NIM, GLM-5.3-Flash via OpenCode, JSONL receipts, httpx, Pillow, FFmpeg, React + Vite (deck), Markdown.

Built With

  • ffmpeg
  • httpx
  • jsonl-receipts
  • mcp-(model-context-protocol)
  • paypal-sandbox-rest-(invoicing-v2)
  • pillow
  • python-3.13
  • react-+-vite-(deck)
Share this project:

Updates

Submission history