SayDO

"Attorney General James Secures $500,000 from Bloomberg News for Deceiving Consumers about Subscriptions" (NY AG). SayDo checks your pricing page against your live PayPal plan.

Comment

Inspiration

"Attorney General James Secures $500,000 from Bloomberg News for Deceiving Consumers about Subscriptions" (New York AG, Nov 2026). A $1.99 intro price rose to $39.99 a month, and customers said they never knew.

Subscription merchants on PayPal face the same trap on a smaller scale. The promises live on the pricing page ("7-day free trial, no setup fees, $9/month"). The real terms live in a PayPal Billing Plan, edited somewhere else. Change one without the other, and the page is now false. Nothing on either side checks that they still agree.

What it does

  • Pre-flight guard. Before a price, setup-fee or tax change reaches PayPal, SayDo shows which sentences on your pricing page it would make false.
  • Audit. SayDo reads the pricing page, finds the PayPal plan IDs inside the page's own subscribe buttons, fetches those plans from PayPal, and marks every promise PASS, FAIL or UNKNOWN. Each verdict shows the exact quote beside the PayPal field and value it was checked against.
  • Continuous monitoring. When a plan changes anywhere (the dashboard, the API, another tool), PayPal's signed webhook makes SayDo re-check the stored promises with no AI call. Verified live on 8 Nov: we raised a plan from $9 to $12 through the PayPal API, the signed event arrived about 12 seconds later, and "$9/month" flipped from PASS to FAIL.
  • Fix with approval. SayDo offers a plan fix through the PayPal API or a page-copy rewrite drafted by the AI. Nothing changes without the merchant's click.
  • An audit console on AG Grid. One row per promise, verdict filters, the PayPal field path, in-place updates that flash when a webhook re-audits, a History tab, and an evidence CSV.

How we built it

pricing page → AI reads it once into typed promises → plan IDs from the page's PayPal buttons → live Billing Plan from PayPal → deterministic checker → PASS / FAIL / UNKNOWN

  • AI reader: Claude via OpenRouter turns copy like "a week free" or "$108 billed yearly, just $9/mo" into typed promises with exact quotes. It never decides a verdict, never calls PayPal and never edits anything.
  • Checker: plain code compares each promise with the plan. Anything a plan can't prove ("cancel anytime") stays UNKNOWN.
  • PayPal (sandbox, live): OAuth, Catalog Products, Billing Plans, Subscriptions, Webhooks with signature verification, and the JS SDK subscribe buttons on the demo store.
  • Stack: Node.js and Express, AG Grid Community 32.3.3, Playwright for headless page rendering, hosted on Render.

Challenges we ran into

  • Messy page copy. Real pages mix add-ons, ranges, one-time fees and labels that look like prices. We fixed the extraction errors real pages exposed and made "uncertain" mean UNKNOWN, never FAIL.
  • Real plans are out of reach. Sandbox credentials can't read a real merchant's plan. So for old-style PayPal Subscribe buttons, SayDo checks the page against the button's own HTML fields, and labels it that way.
  • Determinism over cleverness. A single prompt doing everything was about as accurate, but it flipped 4.2% of its labels between runs. A monitor that re-checks on every webhook needs the same answer every time.

Accomplishments that we're proud of

  • Drift found in the wild. We screened 907 public URLs and found 53 pages on 25 hosts with readable PayPal button fields. Hand-checking found a nonprofit's "Donate $25 Monthly" button that takes one payment and never repeats, and a site whose Subscribe links bill $1 and $1,188.50 against displayed prices of $995 and $1,155.
  • The AI earns its place. On harder held-out pages, the AI reader caught 29 of 30 defects where a regex reader caught 15 of 30, with the same checker behind both.
  • Live PayPal, end to end: a real plan change, a verified signed webhook, and a PASS flipping to FAIL on screen.

What we learned

  • The page and the plan drift for boring reasons. Nobody rereads their own fine print after launch.
  • Verdicts people can trust need receipts. Every SayDo finding is a quote, a PayPal field path and a value.
  • Our evaluation pages are synthetic and the real-page run tests extraction only. We say so in the repo.

What's next for SayDo

  • Run the pre-flight check inside the plan-change path, as a guard any tool editing PayPal plans can call.
  • A "secret shopper" that subscribes through the page's own button in the sandbox and compares the real charges with the promises.
  • Keep audit history in a database instead of server memory.
  • Try it with real subscription merchants.

Built With

Share this project:

Updates

Submission history