Inspiration
Picture the owner of a small rental listings site: five people, a few thousand listings, a hosting bill. Their logs now show one AI bot visit for every 31 human visits (TollBit, Q4 2025), up from one in 200. Those bots download every page, strip the HTML and resell the data. Buyers pay scraping companies for it, and the site that made the data earns nothing.
The owner's options today are bad. Blocking bots earns nothing, and the scrapers come back through proxies. Big publishers sign licensing deals with AI labs, but a directory with 2,000 listings will never get that call. Buyers aren't happy either: scrapers break every time a layout changes, and they return messy HTML, not records.
So we asked: what if the site itself were the cheapest, cleanest way to get its own data, and the owner got paid for it?
What it does
Spigot turns a website into a typed, paid API for AI agents in minutes, without the owner writing code.
- The gate. The owner adds
spigot-gate, a small middleware. Humans and search engines pass through. Declared AI bots and scrapers get an HTTP 402 with a price and a link to the API. The gate decides locally in under a millisecond and fails open, so the site never goes down because of us. - Claude finds the data. Spigot crawls the public pages from the sitemap. Claude reads sample pages from each layout and proposes record types. On our demo site it finds Listing (2,341), Agent (88) and Open House (40). It writes an extractor for each type, tests it on 20 pages and repairs its own mistakes. It also flags personal data, such as agent phone numbers, and hides it by default.
- The owner approves. The owner reviews the proposal, sets a price (e.g. $1.50 per 1,000 records) and publishes. From then on extraction is plain code with zero AI calls per page. Re-crawls become versioned records and a JSON Patch changes feed. If the site's layout drifts, Claude proposes an updated schema for the owner to approve.
- Agents pay with PayPal. An agent builder gets an API key and tops up a prepaid balance with PayPal. A saved card in PayPal Vault enables automatic top-ups within a monthly cap set by the human.
- The agent buys data, not pages. In Claude Code, over our MCP server, the agent asks for "3-bed rentals under $2k that changed today", gets a free quote, then typed JSON. It's charged only after a successful response, and retries with the same idempotency key are never charged twice:
$$\text{charge} = n_{\text{records}} \times \frac{\text{price per 1{,}000}}{1000}$$
- The owner gets paid. Every charge lands in a double-entry ledger, 80% to the publisher and 20% Spigot commission. Payouts go out as PayPal Payouts batches.
- Fair for buyers. A buyer (or their agent, via the MCP
report_bad_recordtool) can flag a bad record. Spigot re-fetches the page, Claude reviews the claim and recommends a credit, and the publisher decides. Unused balance can be refunded to the original PayPal payment.
How we built it
- Backend: Node.js + TypeScript (Fastify, Zod) and Postgres. Each piece of state has one writer: the crawler owns snapshots, the extractor owns records, the ledger owns money.
- Claude API: schema agent. It clusters pages into templates, reads JSON-LD and sample pages, and returns a Zod-validated extractor spec (JSON-LD paths, CSS selectors and allow-listed transforms), never code. It self-checks on 20 pages and repairs failing fields for up to two rounds.
- Claude API: claims triage. It recommends CREDIT or REJECT with a confidence score and never moves money. Untrusted page text and claim text are fenced off as data, not instructions.
- Claude Code + MCP: a Streamable HTTP MCP server with
list_datasets,describe_dataset,quote,search_records,get_changes,get_balance,top_upandreport_bad_record. Spend caps are enforced in code, not in the prompt. - PayPal:
- Orders v2 (
CAPTURE) with the JS SDK v6 button and card fields - Vault plus merchant-initiated orders for auto top-up
- Payouts batches with crash-safe resubmits
- partial Refunds
- Webhooks with signature verification, and polling as backup
- Log in with PayPal for publishers
PayPal-Request-Idon every money call- Only the backend holds PayPal credentials; no model ever does.
- Orders v2 (
- Money: amounts are stored in micros ($1 = 10^6) so a $0.0015 record charge is exact. Every ledger transaction must balance:
$$\sum_{e \,\in\, \text{entries}(tx)} \text{amount}_e = 0$$
- Frontend: React (Vite). The publisher dashboard uses AG Grid, and the records grid builds its columns from the approved schema. The buyer portal shows the PayPal button. An OpenAPI 3.1 spec is generated from the Zod schemas, with typed TypeScript and Python clients.
- Deploy: Docker Compose (Postgres, API and demo site) behind nginx with Let's Encrypt on our own server. SSE and MCP streams run unbuffered.
- Demo site: "Bluebonnet Rentals", a fictional Austin rentals site with the gate installed, plus a script that changes prices so the next crawl produces real changes. Every simulated bot, buyer and call is labelled SIM.
Challenges we ran into
- Micropayments don't work per request. PayPal's fee for a single capture is far more than a $0.0015 record. We switched to a prepaid balance, an internal ledger and batched payouts, so PayPal moves real money a few times and the ledger meters the rest.
- Charging agents fairly. Agents retry. We charge only after a successful response, keyed by idempotency key, and we test that concurrent retries charge exactly once.
- Keeping AI off the hot path. A model call per page would be slow, costly and non-deterministic. Getting Claude to write a validated spec once, test it, repair it and then run it as plain code took many prompt and schema iterations.
- Buyer agents can't hold merchant credentials. PayPal's agent tools run on the merchant's keys, so handing them to a buyer's agent would hand over our merchant account. Our MCP
top_uptool asks the backend to create the order instead. - Sandbox realities. Payouts can't go to the merchant's own account (
SELF_PAY_NOT_ALLOWED), so the demo publisher has a separate sandbox business account. Long-lived SSE and MCP streams also needed proxy buffering turned off.
Accomplishments that we're proud of
- The whole loop works end to end in the PayPal sandbox: crawl → Claude proposes a schema → owner approves and prices it → buyer tops up with PayPal → Claude Code pays for a query over MCP → ledger charge → PayPal Payouts to the publisher.
- Zero AI calls per page or per API request after approval. Claude does the hard thinking once, at setup.
- A ledger that always balances to zero, with PayPal IDs on every money row, and retries that never double-charge or double-pay.
- Fair to both sides: bad-record claims with AI triage, and refunds of unused balance back to PayPal.
- Live and self-hosted: the app, the gated demo site and the MCP server run on our own server with Docker and nginx.
What we learned
- The right customer isn't news sites, which already get licensing deals. It's long-tail sites with structured data (listings, directories, events), where scrapers already prove the demand.
- Facts like prices aren't copyrightable, so the pitch isn't "now it's legal". It's "now it doesn't break, it's fresher, and it's cheaper than scraping".
- PayPal's building blocks (Orders, Vault, Payouts, Refunds and Webhooks) cover a full two-sided marketplace once you put a ledger in the middle.
- An AI is most useful here once, at setup, writing a spec a human approves, not on every request.
What's next for Spigot
- Web Bot Auth: verify signed agents (RFC 9421 drafts) so paying agents are identifiable and stealth scrapers stand out. Today Spigot gates only declared bots; we make paying cheaper and better than evading.
- PayPal multiparty with
platform_fees, so the publisher is the merchant of record and is paid directly (in the demo, Spigot is the merchant). - PayPal Subscriptions for flat-rate change-feed plans.
- A WordPress plugin for the gate, so non-developers can install it in one click.
- Talking to real directory and listings owners and agent builders to price it properly.
Built With
- ag-grid
- claude-api
- claude-code
- docker
- fastify
- log-in-with-paypal
- mcp
- nginx
- node.js
- openapi
- paypal-js-sdk-v6
- paypal-orders-v2
- paypal-payouts
- paypal-refunds
- paypal-vault
- paypal-webhooks
- postgresql
- react
- typescript
- vite
- zod