Inspiration

Peer-to-peer commerce across social media, WhatsApp groups, and online forums is throttled by a fundamental trust deficit. Buyers hesitate to pay upfront for fear of fraud or non-delivery, while sellers worry about shipping valuable goods without guaranteed payment. Existing escrow services are slow, costly, and designed for enterprise transactions rather than everyday peer-to-peer deals.

Inspired by community escrow customs like Indonesian rekber (rekening bersama), we set out to build an autonomous, neutral referee. By combining PayPal sandbox authorization holds with an AI agent that browses the live web for physical proof of delivery, Referee Pay creates a zero-trust escrow where money waits for proof before a single cent moves.

What it does

Referee Pay is an autonomous AI arbiter that coordinates and safeguards peer-to-peer commerce. Sellers describe their deal terms in natural English, and Google Gemini drafts a structured contract with numbered clauses and firm deadlines. The buyer authorizes funds through a seamless PayPal checkout modal (intent=AUTHORIZE), locking the payment safely in escrow without releasing it to the seller.

When tracking is submitted, Referee uses Kernel cloud browser automation to independently inspect carrier portals and record screenshot evidence. A deterministic Policy Engine verifies fulfillment against the contract clauses. If delivery is proven or confirmed, funds are captured and paid out to the seller; if the seller misses the agreed shipping window, the hold is voided automatically to refund the buyer.

How we built it

We built Referee Pay using Next.js 16 (App Router), TypeScript, and Tailwind CSS, backed by Neon serverless PostgreSQL and Drizzle ORM. The financial core runs on @paypal/paypal-server-sdk using two-phase authorization holds, captures, void refunds, and payout disbursements with idempotent request IDs. Google Gemini (gemini-3.5-flash-lite) acts as the contract drafter and arbiter, while Kernel (@onkernel/sdk) executes headless cloud browser runs to capture real-web carrier delivery milestones.

The seller command center utilizes AG Studio and AG Grid for real-time telemetry, risk tracking, and a verified transaction ledger, paired with Bryntum Scheduler and Gantt to visualize fulfillment baselines and arbiter decision markers. API consistency was established using APIMatic to audit PayPal SDK methods and author an OpenAPI 3.1 specification.

Challenges we ran into

The biggest challenge was enforcing financial safety around autonomous AI models. Language models can hallucinate or be misled by prompt injections in tracking notes. We resolved this by adopting a strict architectural principle: the LLM only proposes verdicts, while a pure, deterministic Policy Engine (PE-1 through PE-8) validates clauses and preconditions before any PayPal execution can occur.

We also faced challenges orchestrating headless cloud browsing across varied carrier pages and preserving Gemini tool call thought signatures during multi-turn interactions. We built a custom fetch proxy that preserves reasoning signatures across function calls and encapsulated carrier verification in sandboxed Kernel browser sessions with structured DOM milestone extraction.

Accomplishments that we're proud of

We are proud to have delivered a fully functional, zero-friction escrow lifecycle where buyers can authorize payments via PayPal without creating an account or downloading an app. The system passed 145 automated unit and integration tests covering the state machine, policy engine rules, money calculations, and thought signature integrity.

Our transparent decision log provides explainable commerce: whenever funds move, both parties see an auditable card referencing the exact contract clauses and tamper-proof carrier screenshots. The entire experience runs live in production on Vercel with real-time sandbox execution.

What we learned

We learned that combining generative AI for natural-language understanding with deterministic policy engines for state transitions is the most robust architecture for agentic finance. Generative models excel at interpreting human agreements, but hard rules and idempotent APIs are required when real money is on the line.

We also realized that objective web verification via autonomous cloud browsers bridges the gap between digital agreements and physical-world fulfillment. When an agent can see and record ground truth directly, disputes can be resolved instantly without manual back-and-forth claims.

What's next for Referee Pay

We plan to expand our cloud browser verification beyond standard tracking to direct carrier API integrations with USPS, FedEx, and international postal networks. We also aim to incorporate multimodal vision models to compare pre-shipment condition photos with unboxing media to support fair, automated partial settlements.

Our long-term goal is to release Referee as a Model Context Protocol (MCP) server. This will enable independent buyer and seller AI agents across the web to negotiate, escrow funds, and transact autonomously with built-in arbitration.

Built With

  • ag-grid
  • ag-studio
  • apimatic
  • bryntum
  • drizzle-orm
  • google-gemini
  • kernel
  • neon
  • next.js
  • paypal
  • radix-ui
  • react
  • tailwind-css
  • typescript
  • vercel
Share this project:

Updates

Submission history