Surety — the policy and proof layer for AI agents that move money on PayPal

LLM agents propose. Surety verifies, enforces, and proves. PayPal executes.

Inspiration

AI assistants now handle refunds end-to-end, and the failures are public: Klarna's AI support handled two-thirds of customer chats before the company walked back on model discretion; the FTC fined "AI lawyer" DoNotPay over unsubstantiated claims. Meanwhile PayPal is building agentic commerce (Agent Toolkit, MCP server, Agent Ready) where other parties' agents will request actions on merchant accounts. The missing piece isn't a smarter agent — it's an adjudicator the merchant controls.

What it does

Surety is a deterministic governance layer between customer-facing AI agents and PayPal:

• ARIA, the merchant's support agent, reads customer messages — including adversarial prompt-injection attempts — and proposes refunds through a governed gateway. She never sees PayPal credentials and can never execute a payment herself.
• Surety extracts the claim with a structured-output LLM (Gemini, server-validated schema; customer text is fenced data, never instructions), binds it to the real PayPal capture/order plus fulfillment evidence (shipment tracking and settlement history via the official PayPal Agent Toolkit), and verifies the premise deterministically: a DELIVERED record contradicts "never arrived"; a duplicate-charge claim requires two same-amount captures; a double-refund is caught against settlement history; subjective claims go to humans — honestly.
• The policy engine enforces autonomous caps and per-currency daily limits in exact integer minor units (JPY-style zero-decimal currencies included), with refundable-balance math that cannot be argued around.
• Human review approvals are cryptographically bound to the exact action, claim, evidence, and policy version — a reviewer may exceed daily limits, never the refundable balance.
• Every decision produces a FactPack: a versioned, digest-chained evidence record with full provenance, streamed live and downloadable.

Demonstration highlights

• Agent console: run ARIA over an inbox containing a prompt-injection attempt; every thought, proposal, and verdict streams live.
• Case inbox (AG Grid ledger), FactPack case detail, review queue, policy studio, analytics ("Refunded USD 98 / Blocked USD 649"), verified webhook event log.
• npm run evaluate: an 84-case adversarial suite — 0 false refunds, 0 withheld honest refunds, 100% of injections blocked, p95 < 5 ms — while a modeled ungoverned LLM-discretion baseline pays 38 false refunds ($1,862) on identical inputs.
• npx tsx scripts/sandbox-loop.ts: a real PayPal sandbox order → buyer approval → capture → Surety-adjudicated refund (verified, committed as an artifact).

How we built it

• PayPal: REST (order/capture/refund with idempotency keys, fail-closed reconciliation), @paypal/agent-toolkit for evidence reads (shipment tracking, transactions, refund lookup), signature-verified webhooks triggering reconciliation.
• AI: provider-agnostic structured-output layer (Gemini 3.5 Flash free tier default; Groq/OpenAI adapters); one schema-validated decision per step; deterministic fallback brain so governance never depends on API availability.
• Engineering: Node built-in SQLite (zero native builds), 83 tests including cross-process approval races, React + Vite console with AG Grid themed to a forensic-ledger design system, SSE live trace, Dockerfile + Render blueprint.

Challenges we ran into

• Root-caused a cross-process race where a missing busy-timeout let a concurrent SQLite writer fail silently — fixed and regression-tested.
• Reviewer approvals dead-ended at daily-limit reservations; redesigned approvals to be consumed atomically inside the reservation (humans exceed soft limits, never balances).
• Discovered and fixed a logout bug where the clearing cookie's empty Path silently kept sessions alive.

Accomplishments we're proud of

• Zero false pays with millisecond latency; prompt injection refused at the brain and still safe at the gateway (a compromised agent's proposal still faces the evidence).
• An honest audit trail: every artifact labels what is simulated versus verified against real PayPal sandbox APIs — including a committed record of a real buyer-approved, captured, adjudicated sandbox refund.

What we learned

Deterministic verification is cheap and strong; the hard parts are human-approval semantics and honest provenance. The biggest agent-payment risks are mundane: double execution, replayed approvals, and evidence nobody can audit.

What's next

Dispute acceptance flows, carrier-grade tracking integrations, multi-tenant policy isolation, and an MCP server exposing propose_refund to third-party agents under merchant policy.

Built With

• ag-grid
• docker
• express.js
• gemini
• node.js
• paypal
• paypal-agent-toolkit
• react
• render
• sqlite
• sse
• typescript
• vite