The problem. SiteBefore is an agent-run website service for small businesses: a client approves a quote — a website build, then a monthly care plan — and a billing agent drafts the charges. AI agents are starting to move real money, and their failure mode is quiet: the agent doesn't steal, it errs. It re-charges last month's one-time build fee inside this month's care plan. It bundles an add-on nobody quoted. It pays the right amount to the wrong payee. And in a normal agent stack, nothing checks the draft against the one document that defines what should be charged: the approved quote. A model's confidence is not evidence. We built the missing checkpoint.
What it does. The settlement layer is Claim Court — the verdict gate at the commerce surface of why-agent, our ongoing project whose thesis is that AI claims should be checked against their evidence by something that cannot be sweet-talked — applied to SiteBefore's billing:
- The agent proposes. A payment draft arrives with payee, currency, total, and itemised lines.
- The comparator grades. A deterministic grader (no model in the loop) compares every claim in the draft — amount, currency, payee, each line — against the approved quote, in integer cents, and returns a verdict in a fixed vocabulary:
- CORROBORATED — every claim matches the evidence.
- PARTIAL — something in the draft is outside the evidence (e.g. an unquoted add-on).
- CONFLICTED — the draft contradicts the evidence (wrong amount for this charge occasion, wrong payee).
- The gate decides. CORROBORATED passes the decision-layer screen and the payment runner creates and captures the PayPal order. PARTIAL or CONFLICTED → the payment is held in a human-approval queue. No order is created, nothing is captured, and the agent cannot talk its way past the hold — a human rejects or approves the record, and a corrected payment is drafted and graded fresh.
- It fails closed. No verdict, an unrecognised verdict, or an uncertain screen all end in the queue. The gate itself has no execute path: capture lives only in the payment runner, and only on an allowed verdict. That is the differentiator: not a spending limit or a credential vault, but a verdict — every charge checked against the one document that defines what should be charged, by a gate that cannot execute anything itself.
A note on the AI in this entry. The proposer is an AI billing agent. In the demo, its drafts are reproduced as deterministic fixtures — the exact drafts such an agent produces, fixed so every verdict is reproducible — and the checker is deliberately model-free, because a money gate must not depend on a model's behaviour to be trustworthy.
The demo scenarios (all DEMO CAD amounts — candidate service pricing used as fixtures, labelled DEMO everywhere, not published prices; the recorded run uses the in-memory Orders client — see Honesty notes). The example job throughout is the website redesign + care plan quoted for Shenail Hair Salon & Nail Bar (shenailhairlounge.ca), a real salon in Niagara Falls and SiteBefore's family pilot client — disclosed here, in the demo output, and in the repo README; only its public facts (name, city, website) are used. The scenario is simulated end to end: the quote is the demo's illustrative quote, no payment is a real transaction, and none of it is revenue, traction, or a live customer deployment — the salon has not paid through this system.
- S1 — clean release. The salon's first payment, exactly as quoted: Website Build $900 + Care Plan $99 = $999 CAD → CORROBORATED → captured.
- S2 — the money shot. Month 2 should be Care Plan only ($99), but the agent drafts the full $999 again, repeating the Build line → CONFLICTED → held with zero orders and zero captures. The human rejects it; the corrected $99 re-draft grades CORROBORATED and settles.
- S3 — wrong payee. Right amount, right lines, wrong payee → CONFLICTED → held before any order exists.
- S4 — unquoted add-on. A month-2 charge bundles a $49 AI-Visibility Add-On that was never quoted → PARTIAL → held.
How we built it.
- Deterministic comparator (
comparator.py) — payment claims vs quote evidence → per-claim grades + verdict, each with a human-readable reason. Money is integer cents end to end; no floats near money. - Decision layer (
decision_layer.py) — typed decisions with calibrated confidence, a deterministic rules decider as the gate screen, escalation on low confidence, and the ToolGate whose human-approval queue is the product's centre of gravity. Capture is policy-hardened on top: the capture tool is never allow-listed, and a deny rule hard-blocks any capture whose currency is not the quote's currency before the screen is even consulted. - PayPal integration — two implementations of one Orders interface. The live sandbox path (
paypal_toolkit_client.py) is built on PayPal's official agent toolkit (Apache-2.0, vendored verbatim at a pinned version, provenance invendor/VENDOR.md), using its OAuth transport, order handlers, and payload builder — and it is exercised by 11 tests against the vendored toolkit (the remaining 30 tests cover the comparator, the decision layer, the end-to-end scenarios, and the client contract). One pinned upstream limitation is documented, not hidden: the toolkit'screate_orderhandler is USD-only in the pinned version while this entry is CAD end to end, so order creation routes through its currency-agnostic payload builder. The demo and tests run on an in-memory implementation of the same contract (see "Honesty notes"). - Test posture — the shared core shipped with a full scenario suite before any demo surface existed, and the demo runs the exact scenario builders the tests use. Observed on the shipped code (Python 3.12, dependencies installed): decision layer 8 passed · comparator 10 passed · scenarios S1–S4 end to end 7 passed · client contract 5 passed · toolkit client 11 passed. The gate core is standard-library only; without the two third-party packages the toolkit test file cannot be collected and the other 30 tests still pass.
- Credentials — environment variables only, fail-fast at construction; nothing secret in the repo, ever.
Challenges we ran into. Making "fails closed" true everywhere — every ambiguous path had to end in the human queue, not in execution. CAD vs the toolkit's USD-only order shortcut in the pinned version — solved by routing through the payload builder and documenting the limitation instead of forking silently. Keeping the demo honest — the fixture amounts are candidate prices, not published ones, so the DEMO labelling is part of the product surface, not a footnote.
Accomplishments we're proud of. A payment gate with no execute path that still demos end to end: it releases what matches, holds what doesn't, and shows its reasons. The S2 catch — a plausible, polite, well-formatted agent draft, wrong by $900, stopped before any order existed. A test suite that ran green on the shared core before the demo surface was skinned.
What's next. The same scenes against the live PayPal sandbox — the credential-gated live run is the next verification step; the client itself is implemented and tested (see Honesty notes). A webhook-integrity scenario and signed verdict receipts, so a held payment's why stays auditable months later. Then the real job: this gate becomes the billing checkpoint for SiteBefore itself, whose first pilot job is the Shenail redesign the demo is modelled on, and the comparator + gate core is rail-agnostic, with other payment rails behind the same Orders interface next.
Honesty notes (also stated in the repo README). The live PayPal sandbox client is implemented on PayPal's official agent toolkit and exercised by 11 tests against the vendored toolkit. The recorded demo runs the in-memory implementation of the same Orders interface — the same gate, the same scenario code — so the negative space is provable without credentials: a held payment creates no order and is never captured. The live client fails fast without credentials rather than pretending to work; a credential-gated live-sandbox run is the next verification step, not a missing piece. Judges can run everything in the repo with the included instructions; no hosted deployment is required by the rules. The demo's named client is real — Shenail Hair Salon & Nail Bar, SiteBefore's family pilot client — disclosed as such everywhere it appears; the scenario built around it is simulated, and no copy anywhere in this entry presents it as a paying customer of this system, revenue, or traction.
