TWIG — AI-agent-native data marketplace
Data your agents can't find. Intelligence they can buy.
Inspiration
AI agents are starting to act on their owners' behalf — but the data they need most (API benchmarks, uptime histories, regional performance comparisons) is exactly the data ordinary search can't surface. It's scattered, stale, or undocumented, and there's no store where an agent — or a human — can simply buy it with real guarantees: verified payment, transparent pricing, documented methodology. I built TWIG to be the marketplace I'd want my own agents to shop at.
What it does
TWIG is an AI-agent-native data marketplace with two product lines:
- Ready-made datasets — collected, processed, and validated; buy through PayPal Sandbox and download instantly. Every product page documents schema, sample records, licensing, and collection methodology.
- Custom datasets — describe what you need in plain language. An AI assistant searches the real catalog for matches, or turns your request into a structured specification with a fixed quote, and a background worker collects and delivers the data.
How I built it
- Next.js 16 with Route Handlers as the entire backend — no separate API server. Every request body is Zod-validated.
- Supabase for auth, Postgres (with row-level security), and storage delivery through short-lived signed links.
- A shared TypeScript pricing package that is the single source of truth. Quotes are deterministic — for a benchmark job:
$$P = B_{\text{base}} + n_s \cdot \$18 + n_r \cdot \$24 + n_m \cdot \$12$$
(3 target systems, 1 region, 3 metrics → \$29 + \$54 + \$24 + \$36 = \$143, exactly what the app shows.)
- A Dockerized Python worker that claims jobs via a
claim_job()RPC usingFOR UPDATE SKIP LOCKED, so multiple workers can run concurrently without double-processing. It only ever contacts an allow-list of public test endpoints — job specs contain no URLs at all.
Challenges
The hardest problem was keeping the LLM on a leash. I wanted AI in the buying loop, but an AI that can invent prices, hallucinate datasets, or negotiate its own quotes is a liability, not a feature. So the LLM only writes the spec; pricing, feasibility, and the catalog come exclusively from the pricing engine and the database — with a deterministic rule-based fallback when no AI provider is configured. The result: AI that is auditable and safe to fail.
The second challenge was making payments trustworthy. Prices are computed
server-side (the client can only send IDs), capture happens only on the server,
the captured amount is re-verified against PayPal's payload before granting
access, and unique keys on paypal_capture_id and event IDs make replays and
double-captures harmless.
What I learned
- Designing with a "the client is hostile" mindset changes everything — server-side prices and capture, ownership checks, and RLS as defense-in-depth.
- Idempotency is not an optimization, it's the foundation of anything touching money.
- Constraining an LLM to a narrow role doesn't weaken the product — it's what makes the product credible.
A narrated demo video (recorded with Playwright, voice-over generated by TTS) walks through the full flow: browse → inspect → sign in → ask the assistant → get a deterministic quote → PayPal checkout.
