Delivery Verifier: Pay for work only when it's actually done

Inspiration

Freelance and contractor work breaks down in the same place every time: the client pays before the work is checked, or the contractor finishes and waits for a payment that nobody will release. Platforms that hold money between them usually decide disputes with a person reading a long brief against a messy submission. I wanted to see whether an AI system could handle the checking step, while PayPal handled the money, and whether it could stop short of paying when it wasn't sure.

The idea was simple: a client writes what they want in plain language, an AI turns that into a checklist, the worker submits code, the checklist is checked against it, and the money moves only when the checks pass.

What it does

1. The client writes a brief, and a model turns it into acceptance criteria. Each criterion is marked as a test, a code review, or a manual check. The client approves the list before any work starts.
2. The client funds the case. The app creates a PayPal Orders v2 order with intent: AUTHORIZE, which places a hold on the funds without taking them.
3. The worker submits code. The app runs the test-backed criteria in an isolated process, and sends the code-review criteria to Gemini three times.
4. Routing is deterministic. All criteria pass, so the case is released. A decided failure rejects it and voids the hold. Anything uncertain escalates to a human reviewer, who sees the verdicts and evidence and chooses release or reject.
5. On release, the app captures the held amount and sends a payout to the contractor through PayPal Payouts.

The key design choice is that the model never moves money. It only supplies verdicts. Code applies the release rule, which requires every test to pass and every code-review criterion to be unanimous across three runs.

What we actually verified

We want to be exact here, since the demo depends on it.

• Live in the PayPal sandbox: card-based funding and authorization, the capture (COMPLETED), and the payout submission. The contractor sandbox account received the payout, which we confirmed by checking the balance.
• Live, earlier run: an escalation that was resolved as a reject, which voided the authorization (VOIDED).
• Mock-tested only: the three verdict outputs, and the human approval screen for release and reject. These are covered by automated tests that mock the model and PayPal, not by live runs.
• Not verified: the final payout batch status, which stayed PENDING in our last check. The contractor's receipt came from our own confirmation.

How we built it

• Backend: TypeScript and Express, with SQLite for accounts, cases, and events, and files under out/ for PayPal state and evidence.
• Verification: a sandboxed test runner for the test-backed criteria, and a judge module for the code-review criteria, with the three-run agreement rule.
• PayPal: a sandbox-only client that caches OAuth tokens, sends a stable request ID per operation, and refuses to capture a case twice or after a void.
• Interface: a local web app with a stage-by-stage view of each case, from brief through payout.
• Tests: unit tests for routing, criteria approval, the judge, and the PayPal state guards, all runnable without credentials.

Much of the code was written with AI coding agents. We directed the design and reviewed the results against the test set, but the agents wrote most of the lines.

Challenges

• Sandbox accounts and permissions. Our first business sandbox account was in a country that couldn't process cards, so the card path returned PAYEE_NOT_ENABLED_FOR_CARD_PROCESSING. A new US business account fixed it. Balance reads also needed reporting scopes that the app didn't have at first.
• Buyer approval. The sandbox approval page stalled for us, so the release run was funded through the card route, which skips the approval step. In production the buyer would have to approve the payment themselves, and the demo says so.
• Unclear verdicts. The first release attempt came back unclear because the generated criteria didn't match the test definitions. Fixing the mapping, not the logic, resolved it.
• Path and state bugs. Verification read from the test-set folder, while new cases were created under out/, so new cases crashed with file-not-found errors. Templates now copy a test suite into each new case.
• Sandbox limits. Our build sandbox couldn't bind to a local port, so the route-level HTTP tests are skipped there and run on a developer machine.

What we learned

• Authorization and capture are separate steps with separate failure modes, and a successful API response doesn't prove that money moved. Check the balances.
• Let deterministic code decide anything that moves money. A model's confidence is useful input, but unanimity across repeated runs is easier to explain and audit.
• "Unclear" is a valid outcome, and routing it to a person is the feature, not a failure.
• A demo is only as good as its evidence. Recording the run's IDs and balances matters more than polishing the screens.

What's next

• Accounts and login for real clients and workers, instead of demo accounts.
• Webhooks for payout status, so payments update without a manual refresh.
• Partial captures for multi-milestone projects, and refunds after capture.
• Real notifications when a case needs human review.

Built With

• css
• express.js
• gemini-api
• html
• javascript
• json
• node.js
• npm
• paypalrestapi
• playwright
• sqlite
• typescript