Inspiration
AI agents are starting to buy and sell on our behalf, and the first thing they hit is the oldest problem in commerce: who goes first? An agent that pays another agent, a freelancer or a seller on a chat app pays blind. For millions of small sellers in emerging markets, "I paid and received nothing" is the reason buyers never deal with them at all. Escrow solves this, but today it is manual, slow and opaque. We wanted trust that software can enforce, and explain.
What it does
PACT is programmable escrow for agents and people. PayPal holds the money, Meta Muse checks the proof, and funds move milestone by milestone.
- Say the deal. "50 embroidered t-shirts, 30% at production, 70% at delivery." Muse turns it into a contract with milestones and criteria a photo can actually prove, or asks questions instead of guessing.
- PayPal holds the money. The buyer approves once in PayPal: an Orders v2 order with
intent: AUTHORIZE, so funds are held, not charged. That single human approval is the spending cap every agent works inside. - Evidence, verified. Deterministic checks run first (file type, size, evidence reused from another deal, deadline). Then Muse describes the evidence without seeing the criteria, and a second Muse step judges that description against each criterion. Low confidence never releases money.
- Programmable release. Each approved milestone is a partial capture of the authorization, the last one a final capture. What can no longer be earned is voided, a dispute decided for the buyer becomes a refund, and subcontractors can be paid with Payouts.
- Proof filmed live, not uploaded. For valuable goods, the buyer can switch uploads off. The seller films the goods for 20 seconds through a one-time link while PACT, at the moment recording starts, draws three random instructions (lift an item, show two fingers, turn it over). Muse checks each one on frames taken at those exact seconds, then judges the goods. A stock photo or an image prepared in advance cannot answer instructions nobody knew in advance.
- Agents on a leash their owner holds. Each agent key carries a mandate: per-pact limit, budget across open pacts, allowed categories, and an amount above which its human owner must approve. It is enforced on the server inside the MCP tools, so no prompt can talk the agent past it.
- Transparent arbitration. Disputes go to an AI Arbiter that cites the evidence and decides release, partial or refund, with a human appeal button on every decision.
- For agents. An MCP server exposes
create_pact,fund_pact,submit_evidence,verify_milestone,release_funds,open_dispute,get_reputationand more. In the demo, two Muse agents play buyer and seller with no human scripting. - For operators. The PACT Control Tower, built on AG Studio, shows the ledger live: KPI tiles, Escrow Vault, Milestone Timeline, Verdict Card, an AG Grid reputation leaderboard, a Money Flow Sankey, PayPal reconciliation and an Anomaly Radar. Two custom agents live in Studio's Agent Framework: the PACT Auditor explains any decision from the evidence and builds widgets by delegating to Studio's built-in agents, and the PACT Sentinel, an agent with its own loop, flags risks with nine deterministic rules and has Studio's Widget agent draw the chart.
- Auditable. Every state change and AI decision is appended to a SHA-256 hash chain that anyone can recompute.
Reputation, without punishing newcomers
Each rate uses a Beta prior, so a new seller starts at about 83.5/100 instead of 0 (or a lucky 100):
$$\text{rate} = \frac{k + a}{n + a + b}$$
$$\text{score} = 100\,\big(0.40\,c + 0.25\,t + 0.20\,(1-d) + 0.15\,(1-r)\big)$$
where $c$, $t$, $d$, $r$ are the smoothed completion, on-time, dispute and refund rates.
How we built it
TypeScript end to end, in a monorepo that runs with npm install && npm run dev: no Docker, no database server.
- PayPal (sandbox): OAuth2, Orders v2
AUTHORIZE, Payments v2 partial and final captures, void, reauthorize, refunds, Payouts, webhook signature verification, and Transaction Search plus balances for reconciliation. Every write carries a deterministicPayPal-Request-Id, with bounded retries and typed errors. A reconciliation poller covers what webhooks cannot reach on localhost; a job handles the 3-day honor period and 29-day validity. - Meta Muse (Meta Model API, OpenAI-compatible): Contract Parser, two-step multimodal Verifier, Arbiter, live-video instruction check, the two MCP agents and the Studio agents. Every output is a Zod schema compiled to strict JSON Schema. A record/replay cache lets judges run the whole demo without a key.
- AG Studio and AG Grid: a PACT theme (light and dark) built on Studio's theming API, a matching AG Grid theme, eleven custom widgets that adapt to their size, Studio charts and filters on five pages, live data over Server-Sent Events, drill-through, and the Agent Framework (a custom LLM agent plus a
clientToolRunneragent that delegates to Studio's Widget agent). - Core: an XState state machine with guarded transitions, SQLite with Drizzle, Fastify, OpenAPI 3.1, Vitest, Playwright and GitHub Actions.
Challenges we ran into
- An agent cannot click "approve" in PayPal, and should not. We made the human approval the feature: agents act inside an authorization the human capped.
- Vision models can be wrong. We split verification into describe blind, then judge, added deterministic pre-checks, and wrapped the verdict in a policy that turns low confidence into "needs more evidence". Then we measured it on 36 labelled cases, including tampered photos and a sign telling the AI to approve: 0 of 26 bad submissions released money, and all 10 good ones were paid. Building that set also exposed a real bug (two different photos with the same file name could share a cached answer), now fixed.
- Real authorization lifetimes. Testing a 4-day-old sandbox authorization showed that a reauthorization keeps the original expiry and cannot itself be voided, so PACT now always voids the original.
- Bringing our own model into AG Studio. Studio's tools use deep, recursive JSON Schemas that Meta's API rejects; our server-side proxy rewrites them into bounded schemas while Studio still validates every argument.
- A rules engine as a first-class agent. Studio's
clientToolRunnerlet the Sentinel produce its own turns and delegate chart-building to Studio's Widget agent, without being an LLM. - Determinism. Recorded answers break as soon as ids or dates change, so we normalise volatile values in cache keys and shift dates on replay.
Accomplishments that we're proud of
- A real money flow in the PayPal sandbox: authorize once, 30% partial capture, 70% final capture, refund, void, a dispute settled at 60%, and a Payouts batch.
- Two Muse agents play the whole deal over MCP. The seller agent sends the wrong photo, reads the Verifier's reasoning and corrects itself.
- Live video proof tested end to end with a real webcam, real Muse checks and real sandbox captures.
- One chat question travels through three agents (PACT Auditor → PACT Sentinel → Studio's Widget agent) and leaves a pinned finding and a new chart on the dashboard.
What we learned
Trust is a product feature you can engineer: limits set by humans, decisions made by software, explanations for both sides, and a log nobody can quietly rewrite.
What's next for PACT
- Weight Arbiter decisions into reputation, and verify party identities.
- Production PayPal onboarding for marketplaces, and more currencies.
Built with Meta Muse as the runtime AI. Claude Code was used as a development tool only. Without PayPal credentials, a clearly labelled SIMULATED adapter stands in for PayPal. The demo video replays recorded real Muse outputs, and its narration is AI text-to-speech.
Built With
- ag-grid
- ag-studio
- drizzle-orm
- fastify
- meta-model-api
- meta-muse
- model-context-protocol
- motion
- next.js
- node.js
- openapi
- paypal
- paypal-js-sdk
- paypal-payouts
- paypal-rest-api
- playwright
- react
- server-sent-events
- sqlite
- tailwind-css
- typescript
- vitest
- xstate
- zod