Seen: AI live shopping with PayPal

If it's on screen, it's shoppable. AI turns any live video into PayPal checkout and pays the creator instantly with PayPal Payouts.

Comment

Inspiration

Live shopping is huge, but outside closed apps like TikTok Shop it still runs on "link in bio". A viewer sees a lamp they love on a creator's stream, scrolls for the link, and the impulse is gone. Creators who do earn affiliate money wait weeks for it, spread across networks, with no clear view of what sold.

I asked a simple question: what if anything visible on screen could be bought right there, and the creator got paid in minutes instead of weeks? PayPal already has both halves: trusted checkout for the viewer and Payouts for the creator. AI could be the glue between them.

What it does

Seen watches a creator's video or live stream and recognizes the products on screen. No tagging, no links, no setup.

  1. The creator goes live with a video, a screen share or their own stream URL.
  2. AI spots the products (a lamp, a mug, a laptop) and matches each one to a real partner store's catalog in seconds.
  3. Viewers watch the stream live and see every product beside the video, with Buy now. PayPal checkout opens on top, and the video keeps playing.
  4. The sale is attributed to the creator, a commission is recorded, and Pay me now sends it with PayPal Payouts.

On top of that:

  • A viewer shopping agent. It answers "what's on screen?" or "is the mug cheaper elsewhere?" with real price comparisons, and prepares checkout. It never pays on its own: the viewer always approves in PayPal.
  • A creator assistant, Ask Seen. It answers "How much did I earn this week?" from the sales ledger and looks things up in PayPal with the PayPal Agent Toolkit (read-only).
  • Live earnings (AG Grid): new sales slide in and glow, and totals update, with no reload.
  • Brand slots (Bryntum): brands request a time slot to get their product featured on a stream, and the creator approves.
  • Sale and payout alerts through Zapier, and a shareable AI recap of each stream.

How I built it

  • App: Next.js 16 (App Router) + React 19 + TypeScript (strict), Tailwind and shadcn, Prisma 7 on Neon Postgres with pgvector.
  • Vision pipeline:
    • The creator's browser samples frames and sends one only when a perceptual hash (dHash) says the picture changed.
    • Gemini returns the products through a strict JSON schema, validated with zod.
    • Guardrails drop people and blocked categories, and treat any text in a frame as data, not instructions.
    • A tracker keeps cards from flickering: an item has to be seen twice, unless it's large and clearly visible.
    • Backups: OpenRouter's free vision models step in when Gemini is busy, and a saved-results cache lets the demo video run with zero AI quota.
  • Matching: embeddings plus text search (Postgres + pgvector, with optional Elastic hybrid search) find the partner product. Channel3 adds similar items from the open web and a price comparison.
  • Live delivery:
    • Products are pushed to viewers with Server-Sent Events.
    • Video goes browser to browser with WebRTC, so there's no media server to pay for. If a direct connection fails, viewers fall back to live frame snapshots.
  • PayPal:
    • Orders v2 + JS SDK v6 for checkout (PayPal button and guest card), plus Payouts, Refunds and verified webhooks.
    • Every call is idempotent (PayPal-Request-Id, sender_batch_id).
    • The browser's capture and PayPal's webhook both run through one state machine: each transition is applied, noop or rejected, and only applied triggers side effects. A sale can never be counted twice.
  • Money: everything is in integer minor units (cents), computed on the server from the database, never by the AI:

$$ \text{commission} = \operatorname{round}\left( r_{\text{creator}} \times \text{captured item total} \right) \ \text{cents} $$

A commission moves $\text{PENDING} \rightarrow \text{ELIGIBLE} \rightarrow \text{PAID}$, and to $\text{REVERSED}$ if the sale is refunded.

  • Sponsors: KERNEL can watch a creator's public stream URL in a cloud browser, and an Astropods agent can drive it. A Render Blueprint is ready for one-click deploy.
  • Testing: 132 unit tests, database tests for money flows (capture/webhook races, refunds, payouts), Playwright end-to-end tests, and a Postman collection.

The whole project runs for \$0. Every service is free or free-tier, and a quota guard drops to saver mode, then cached results, then "unavailable", before any limit is hit. Checkout always keeps working.

Challenges I ran into

  • Keeping AI away from the money. The AI only identifies what's on screen. Prices, totals, commission rates and payout amounts come from the database, enforced in code, not in a prompt. Even the shopping agent's "this is cheaper" claims are checked in code. I caught it once calling a similar mug "the same product", and once calling a more expensive one "the cheapest", and fixed both.
  • Two paths reporting one payment. The browser capture and the PayPal webhook arrive in any order, sometimes at the same moment. One state machine with idempotent transitions turned that race into a non-issue, and tests reproduce it.
  • Flicker and duplicates. Vision models describe the same mug slightly differently from one frame to the next. Stabilizing detections (two sightings, embedding dedupe, grouping by item type) took several rounds.
  • Live video for \$0. Media servers cost money, so I went peer-to-peer with WebRTC, with a snapshot fallback for networks that block it.
  • Free tiers sleep. The free database pauses when idle, so I added safe retries and a calm "Reconnecting…" screen instead of a crash.
  • Sandbox realities. PayPal sandbox payouts needed the right account setup (a US business account) before captures and Payouts worked end to end.

Accomplishments that I'm proud of

  • The full loop works for real in PayPal sandbox: frame → detection → product card → checkout → commission → payout, from a video frame to a payout in the creator's account.
  • Money is safe by design: idempotent PayPal calls, verified webhooks, integer cents, an audit timeline for every money event, and AI that never touches amounts.
  • Graceful degradation: every sponsor integration can be switched off, and the core loop still works.
  • Built solo, for \$0: every service is free or free-tier.

What I learned

  • Agentic commerce is mostly about boundaries: what the agent may do, what it must ask for, and what it must never decide.
  • With PayPal, idempotency and webhooks are the backbone of a trustworthy payment app, not an extra.
  • Structured output plus validation turns a vision model into a dependable component, as long as the AI result is never treated as final.

What's next for Seen: AI live shopping with PayPal

  • Multiparty payments: onboard partner stores with PayPal's marketplace solution, so funds go straight to each merchant and Seen takes a platform fee.
  • Paid brand slots: brands pay the creator's slot fee through PayPal when a slot is approved.
  • More ways in: native integrations with streaming platforms, Venmo checkout for US viewers, and products mentioned out loud on the stream.

Built With

Share this project:

Updates

Submission history