-
-
Said & Done. The till that listens.
-
Paid: the invoice moved UNPAID to PAID, with the measured time from sentence to money
-
The spoken sale became a $22 PayPal invoice: QR for the buyer, tool calls and timings on the proof card
-
Books opens on the last market day when today is empty; cash and PayPal in one ledger
Inspiration
The sale is gone the moment a buyer says "I only have a card" while your hands are in the honey, and the evening is gone when you sit down to turn a cash box, three phone payments and a scrap of paper into one number.
Card readers answer that with hardware, fees and a catalog you have to set up by tapping. So the person behind a market or craft-fair table stays on cash, or on a payment app they use with one hand while serving with the other. USDA's National Farmers Market Directory listed 8,771 markets in its last published national count (2019, https://www.ams.usda.gov/sites/default/files/media/NationalCountofFarmersMarketDirectoryListings082019.pdf). That is a count of markets, not of sellers, and we make no claim about how many sellers lose sales this way.
What it does
Said & Done is the seller's counter that listens. The seller says "two jars of honey and the big candle, she pays by PayPal". Five to six seconds after the sentence, a full-screen QR code is on the laptop. The buyer scans it with their own phone, pays in PayPal, and the seller's screen turns green: "$22 Paid." The agent says "Paid. Twenty-two dollars." and goes back to listening. Nobody typed anything and the buyer installed nothing.
The same microphone handles the rest of a market day:
- Cash: "Three small candles, cash." The invoice is closed as a cash payment and lands in the same book as the PayPal sales.
- Pay later: "One gift box, she pays later, her name is Anna." A PayPal invoice is emailed, the Owes tab lists who still owes, and "remind everyone who still owes" sends the reminders.
- Corrections by voice: "No, three" changes a misheard quantity while the chips are on screen.
- Catalog by voice: "I sell lavender sachets for four dollars, people call them smellies" creates a PayPal catalog product.
- The day, spoken and written: "How did today go?" answers with sales, cash versus PayPal and the top item. "What should I bring next week?" compares with the last market day. The Books page shows the same numbers in an AG Grid ledger, by item and by payment method.
Every sale made live is a real PayPal sandbox payment. The proof card on the QR and PAID screens lists the PayPal tool calls with their timings, shows the invoice moving from UNPAID to PAID, and reads out the measured time from the end of the spoken sentence to the paid invoice, buyer login included.
How we built it
Flow: speech (browser Web Speech API) → Gemini parses the sentence against the catalog aliases → code prices every line → PayPal Agent Toolkit creates, shares and QR-codes the invoice → the app polls the invoice every 3 s until it reads PAID.
| Step | What runs |
|---|---|
| Sale understood | Gemini (gemini-3.7-flash, fallback gemini-3.5-flash raced after 4 s) via the Vercel AI SDK returns structured items and quantities. Code, not the model, prices every line and recomputes the total on the server. |
| Invoice | PayPal Agent Toolkit tools create_invoice, send_invoice (share link, no email), generate_invoice_qr_code |
| Payment detected | get_invoice, polled every 3 s until PAID |
| Cash | record_payment_for_invoice (method CASH) |
| Pay later and reminders | send_invoice with email, send_invoice_reminder, cancel_sent_invoice for "never mind" |
| Catalog by voice | create_product, list_products |
| Books | list_invoices and get_invoice, shown with AG Grid Community |
The tools are called directly, with no model in the money path. Stack: Next.js 15 (App Router, TypeScript) on Vercel. Repo: 23 commits made during the event, 168 passing unit tests, and an 18-sentence parser eval against Gemini (18 of 18 when last run on 2026-11-10).
Not used, on purpose: list_transactions (returns 403 for this sandbox app) and get_merchant_insights (not supported in the sandbox). End-of-day numbers are computed from invoice data. We do not claim reporting.
AI use, stated plainly: Gemini runs inside the product (parsing the sale). The code was also written with AI coding assistance.
What did not work
- Direct REST calls. We planned PayPal REST calls for invoices and found the Agent Toolkit tools did the whole job, so there are none.
- Reporting tools.
list_transactionsreturned 403 andget_merchant_insightsis unsupported in the sandbox. The books are computed from invoices instead, and the README says so. - Gemini 2.5.
gemini-2.5-flashis closed to new API keys, so the parser runs on 3.7 with 3.5 as fallback. - A microphone indicator over the QR code. The first voice test drew the indicator across the QR so it could not be scanned. It moved, and the QR now sits on a plain white card.
- "Okay" as a confirmation word. A customer's "okay" could have fired an invoice. Only "yes", "yeah", "go ahead" and "that's right" confirm a sale now.
- Bystander speech. At a stall the seller talks to customers all day, and the first version answered "I didn't catch that" to chatter. The counter now ignores non-sale speech and shows it faintly for a few seconds.
- Webhooks. We poll instead; the app is a demo-scale counter, and polling resumes after a dropped connection without a reload (tested with Wi-Fi off for about 100 s).
Challenges we ran into
The sale has to feel instant even though three PayPal calls sit behind "yes". Warm, a sentence reaches the QR in about 5 to 6 seconds. Misheard quantities are the common failure, so every chip is editable by voice or tap and nothing restarts. Money correctness took four review rounds: sub-cent prices, a cancel racing a cash mark, and a reused sale key were all found and fixed with tests.
Accomplishments that we're proud of
A stranger's phone pays an invoice that was a spoken sentence a few seconds earlier, and the screen turns green without a reload. The product also keeps its honesty on screen: curated history is labelled "Curated history" in the ledger, and "All days" shows every invoice, test invoices included.
What we learned
The hard part of a voice agent near money is what it must refuse to do: not price in the model, not confirm on "okay", not answer chatter. We also learned how much of a payments flow the Agent Toolkit already covers when you call its tools directly.
What's next
Webhooks instead of polling, queueing sales when the stall has no signal, receipts by SMS, importing an existing catalog from a card-reader app.
Built With
- ag-grid
- gemini
- next.js
- paypal-agent-toolkit
- paypal-invoicing-api
- typescript
- vercel
- vercel-ai-sdk
- web-speech-api
