Escrow Agent — AI Referee for PayPal Escrow

An AI payment-operations agent that holds PayPal funds in escrow and releases them only when it verifies the deal conditions were met — and shows its work for every decision.

Comment

Inspiration

Marketplace escrow today is either a human reviewing screenshots or a timer that pays out whether or not the goods arrived. PayPal's authorize/capture/void flow is already an escrow primitive; what's missing is a referee you can trust to pull the trigger. The design question we kept asking: how much should each layer of an AI system be trusted with money? Our answer became the thesis — AI you can trust with money, because it shows its work, and because every layer of it is only trusted as far as it was measured.

PayPal is publicly building for agentic commerce at scale — its own platform messaging describes thousands of engineers creating agentic capabilities for more than 400 million users. But when an agent can spend, something still has to decide whether the money should actually move. That's the governed layer this project builds: an escrow referee whose authority is bounded by measurement — a deterministic kernel that owns every hard call, and an AI sensor that is advisory only, never authority.

What it does

  • Anyone can start an agreement: one form captures what is being bought or built, who the parties are, the total, and the structure — a single lump, or a list of milestones, each with plain-words completion requirements, its share of the money (validated to sum to the total), and its due date.
  • Sharing is a link. The proposer gets a buyer link and a seller link, each gated by its own unguessable token; opening your link is your identity for the demo. The counterparty sees the whole agreement — milestones, dates, and what happens at each outcome — before accepting. Accepting locks the terms; declining closes the deal cleanly with no money ever moved; an unaccepted proposal can be withdrawn.
  • A funded deal is a PayPal authorization that locks the buyer's money (capture = release to seller, void = refund to buyer, partial capture + void of the remainder = a measured split between the two). Each milestone settles on its own: a failed milestone refunds its share without unwinding the milestones that already settled.
  • A deterministic Tier 0 checklist verifies the conditions that can actually be checked: pickup scan, delivery, ZIP match, amount match, dispute state. Hard fails short-circuit to REFUND, decisive passes to RELEASE, condition disputes to INVESTIGATE. Most cases end here, at $0 AI cost.
  • A Tier 1 shadow sensor (Laya decision forest, local, $0/decision) answers atomic factual questions about unsettled cases — per checklist item, not just per decision. Every item renders with its evidence, Tier 0's call, and Laya's approve/disapprove beside it, with an agreement marker; disagreement escalates that milestone to INVESTIGATE. It can flag. It can never move money.
  • Tier 2 is reserved for genuinely ambiguous, higher-value cases only: an AI layer reviews the documents on hand for the case and escalates to a human only if it is still uncertain.
  • Every decision — verdict, tier, evidence, reasoning, cost, shadow opinion — lands in an append-only audit log, and either party's link reopens the complete record: agreement, acceptances, funding, evidence, and every decision with its reasoning. A live dashboard shows deal states with their agreed completion dates, the event feed, and cost-per-decision.
  • An authorization-expiry watch guards the hold itself: PayPal fully honors an authorization for ~3 days and expires it at 29, so monitoring flags an aging hold in the audit feed at 20 days and escalates the deal at 25. An escrow must never discover an empty hold at day 30.
  • The same kernel grades other kinds of delivered work. A code-delivery domain grades a delivered GitHub repo against the client's requirements list (files, non-stub functions verified by AST, the test suite, README sections) and settles at the measured completion fraction. A PR-grading domain grades a pull request against the requirements in the issue it claims to close — changed files, check runs, the PR's own checklist. The requirements list is the agreement and the audit list in one artifact — the same frozen list at funding and at grading.

How we built it

Python. PayPal sandbox Orders API (authorize / capture / void) behind paypal_client.py; a deal lifecycle state machine (PROPOSED → FUNDED → AWAITING_APPROVAL → MONITORING → DECIDED); the checklist array in checklist.py; the tiered decision engine; Laya (PyPI laya==0.4.14, checkpoint convaiinnovations/laya/typed-decisions) running locally as the shadow forest; a scripted tracking simulator and three demo deals ($500 camera, $300 keyboard, $1,200 laptop — the laptop structured as a $900 delivery milestone due Nov 5 plus a $300 condition milestone due Nov 9); pytest suites. The web surface — operator dashboard, party pages, the propose form, and the /grade workbench — shares one design system (Atlas Glass: dark shell, frosted cards, verdict pills). Muse was the development pair: architecture review, test design, and the measurement harnesses that decided Laya's role. Public repo: github.com/iacosta3994/paypal-escrow-agent (Apache License 2.0).

Challenges we ran into

  1. We benchmarked our own AI judge and demoted it. The Laya forest was first built as the adjudicator. It failed the adjudication checklist (2026-10-09), and a later complexity ladder (94.7% per-question accuracy) showed deterministic degradation on mixed-valence state blocks. So it was demoted to a shadow sensor with a deterministic aggregator — trusted exactly as far as it was measured, no further.
  2. Our pipeline, not the model. Two cases scored perfect answers but landed on INVESTIGATE because the forest re-parsed rendered state text instead of using the facts it already had. Fixed 2026-10-24 by passing build_state() facts through directly; both cases flipped to their ground-truth RELEASE, accuracy unchanged.
  3. "End-to-end" wasn't. When we finally tested the not-ready cases against the real sandbox (2026-10-24), the judging brain worked and the money leg had never moved a dollar: the buyer-approval step was never wired (422 ORDER_NOT_APPROVED), fund() discarded the authorization ID so settle() could flip a deal to RELEASED without capturing, settle() had no state guard (a probe re-settled a RELEASED deal as REFUNDED), and dispute state was hardcoded off. All four were fixed and covered by tests that assert PayPal-side state, not local flags — and on 2026-10-25 the last unverified leg closed: with a human-speed buyer approval, the full $300 authorization → capture path ran against the real sandbox, and a SPLIT settlement ($150 captured, remainder voided via partial capture) was proven live the same way.

Accomplishments we're proud of

  • 432 tests collected; the unattended suite runs 431 passed / 1 skipped (freeze-gate run, 2026-11-09), and the one skip is honest: completing a buyer approval has no PayPal API — it needs a human click in a browser. That leg was verified in manual mode on 2026-10-25 (full suite 368/368 at that build, real sandbox orders approved → authorized → captured/voided; evidence: order 9AY42703L3607051D → authorization 6N471818HL119145T → VOIDED).
  • Split settlement is real money math, proven live: a $300 escrow split captures $150 and voids the rest via partial capture — the settlement follows the measured fraction, not a coin flip. The milestone smoke run walks the same math per milestone: the laptop's $900 delivery milestone released clean, its disputed $300 condition milestone split $150/$150, and the deal closed SPLIT_SETTLED at $1,050/$150.
  • The shadow sensor is measured, not vibes: on a frozen 570-question hostile ladder the forest holds 0.947 pooled accuracy (540/570, at its regression baseline in every recorded gate run, including the 2026-11-09 freeze gate), and the ladder is wired in as a gate — it has already caught one of our own changes regressing the sensor and sent it back. Around the money path sit a hygiene gate pinning the sensor's input shape and a golden corpus of messy real-world deals (partial shipments, ZIP typos, carrier exception strings, cent-level mismatches).
  • † Webhook deliveries are signature-verified before they are trusted: with a webhook id configured, every delivery is checked against PayPal's verify-webhook-signature API and failures are rejected with no side effects — proven live against the sandbox API with forged deliveries over a public tunnel (both rejected, 400).

Limitations (summary — full text in the README)

The money rail is a card hold, not production escrow: production would capture into a platform account at funding and disburse via the Payouts API; the adjudication core is rail-agnostic, so that swap doesn't touch the judgment. Party identity is demo-grade: the shareable links carry unguessable per-party tokens, and production would use real accounts. Milestone refunds are decided per milestone immediately, but the void executes when the authorization closes — PayPal allows one void per authorization. The forest's blind spots are measured and contained (mixed-valence both-NO reads, numeric-line contamination, a YES lean on UNKNOWN facts — Tier 0 owns the hard fails, a hygiene gate pins its input shape, and its verdict is shadow-only). The accuracy corpus is self-authored: 100% on our cases is a claim about guardrails, not the world. Buyer approval has no PayPal API and stays a human click — an unattended suite run therefore reports that one test as skipped, not passed, and says why.

What we learned

Trust in an AI money system isn't a property of the model; it's a property of the measurement around it. A small model that is benchmarked, demoted, and audit-logged is worth more than a big one with a vibes-based mandate — and "tested end-to-end" means the money actually moved, verified on the payment provider's side.

What's next

The road from this build to a service is named, not hidden: capture-at-funding with Payouts-API disbursement (the production rail the Limitations section describes), ACH/wire funding for high-value deals so a card dispute can't reverse a verdict, double-entry ledger accounting with daily reconciliation of held funds, real accounts with identity verification in place of demo token links, a dispute operations console behind the INVESTIGATE state, and Tier 2 escalation wired — in shadow first, benchmarked on real cases — for the ambiguous high-value cases the sensor flags. Nearer term: real carrier tracking APIs in place of the simulator.

Built With

Share this project:

Updates

Submission history