Judge Pay

AI drafts the checks and does the work. PayPal holds the money. Only the agreed checks release it: passing work is captured and paid out, failing work voids the hold.

Comment

Inspiration

AI agents can already do paid work, and paying them is getting easy. The hard part is deciding when the money should move. A buyer either trusts the agent's promise or checks every delivery by hand. We wanted acceptance to be part of the payment itself: agreed before the work starts, enforced by PayPal, and never decided by the same AI that did the work.

What it does

Judge Pay is an acceptance and payment layer for work bought from AI agents or people.

  1. The buyer describes a task. GPT-4.1 Mini, through OpenRouter, drafts objective acceptance checks and lists what no program can check. The buyer reads them in plain language and can edit every one.
  2. The buyer approves a PayPal authorization through PayPal Buttons. Nothing is captured yet. A fingerprint (hash) of the checks is written into the PayPal order's custom_id and verified before the hold is accepted, so the terms cannot change after funding.
  3. The worker opens a private job link and delivers, either by hand or with the AI worker, which drafts the work, runs the same checks on its own draft and revises, for up to three drafts, submitting only a draft that passes.
  4. Judge Protocol's deterministic engine runs the agreed checks on the delivered bytes. Pass: Judge Pay captures the authorization and sends the worker a PayPal payout. Fail, or no delivery by the deadline: the authorization is voided and the buyer is never charged.
  5. Every judged job produces a receipt that anyone holding the delivered file can recompute in the browser, and an AG Grid dashboard shows every job, score, PayPal record and exception.

The demo task reconciles three refund receipts into JSON. A delivery must have the exact total, exactly the three allowed receipt IDs and only the agreed fields. In the video, the AI worker's delivery passes and is paid; a delivery with one made-up receipt ID (RCPT-1049) is rejected and its hold is voided.

How we built it

  • PayPal: Orders v2 with intent AUTHORIZE, authorization capture and void, PayPal Payouts, and the PayPal JavaScript SDK Buttons, all in the real job lifecycle. Money-moving calls use idempotency keys and recovery reads, so a retry or a lost response never double-captures or double-pays.
  • AI: OpenRouter with openai/gpt-4.1-mini and strict structured output, in two roles: drafting the checks, and doing the work in a draft, check, revise loop. When a request names IDs (like RCPT-1042) and the draft does not pin them, the drafter is sent back to pin them. The model never decides payment.
  • Judge Protocol engine: our open-source deterministic checker (JSON shape, required content, length, checksum), the same code in the server and the browser verifier.
  • App: Node.js API with an explicit job state machine; plain HTML/CSS/JS pages; AG Grid Community for operations; hosted on Vercel with jobs in a private Vercel Blob store using ETag-checked writes.
  • 92 automated tests for Judge Pay, plus the engine's own suite.

Challenges we ran into

The hard part was not calling a payment API. It was keeping the terms fixed between AI drafting and payment, recovering safely from uncertain payment responses, and making AI drafts strict enough to trust: a live draft once let any receipt ID through, which is why the drafter now has to pin every ID the request names. On hosting, Vercel Blob returns a weak ETag on compressed reads, which broke our conditional writes until we found and tested it.

Accomplishments that we're proud of

Both outcomes run end to end on the hosted sandbox through real PayPal Buttons, and every record was read back independently through PayPal's REST API:

  • Pass: job 1793811661767544, order 2MM41002GF227060G, authorization 5WH57852RH471421A captured, capture 7RG09555JS853171C, payout batch PLH83VYQ5AQCN SUCCESS ($4.75 to the worker).
  • Fail: job 1793813696038357, order 8TV75356K2617373N, authorization 9MJ4895122309292K voided, no capture, no payout.

What we learned

AI is most useful before and during the work, not as the judge of its own payment. A few deterministic checks, reviewed before funding, make a much clearer trust boundary than an AI score. Payment integration also needs failure recovery and evidence, not just a successful happy path.

What's next

Worker identity and a delivery credential only the worker holds (today the buyer receives the worker link and passes it on), signed receipts, more objective check types, payout webhooks, notifications, and a transactional database with per-account limits. Subjective work belongs in a separate review and dispute flow, not behind a pretend deterministic score.

Honest scope

Judge Pay is a sandbox prototype, not a production escrow service. Deterministic checks verify structure, exact values, required content and length; they cannot prove truth, taste or the authenticity of external records. It builds on our existing open-source Judge Protocol engine; everything in Judge Pay (the PayPal integration, the web app, the AI drafting and worker, the receipt checker, the dashboard and the hosting) was built during the hackathon, from October 22 to November 4, 2026.

Built With

Share this project:

Updates

Submission history