Paygenti

The AI cashier for street vendors: say the price, the customer taps or scans to pay with PayPal, and your phone or counter box says "¡Pagado!"

Comment

Inspiration

A street vendor works with her hands full. She can't stop to type on a card terminal, a "payment sent" screenshot is easy to fake, and when most of her income is cash she has no record a lender trusts. Our demo vendor is Rosa, who sells tamales and atole in Oaxaca. We wanted a cashier she could run by voice, that only celebrates money PayPal actually captured, and that turns her sales into a history she owns.

What it does

  • Voice to cart. Rosa holds the mic and says "two mole tamales and an atole". Claude turns it into a cart from her own catalog, and every item and price is checked against that catalog, so the model can't invent either. Without an API key, a rule-based parser does the same job.
  • Tap or scan to pay. The customer taps an NFC sticker or scans a QR and pays on their own phone with PayPal's JS SDK v6 checkout: PayPal and card, plus Venmo and Pay Later where PayPal says the buyer is eligible.
  • "¡Pagado! Ninety-five pesos". Only after our server captures the order with PayPal does the vendor's phone turn green and say the amount out loud: a chime and recorded voice clips, in Spanish or English. An old Android phone on the counter can be the soundbox. A fake screenshot stays silent.
  • A voice box on the counter. An ESP32-S3 box we built this week (microphone, speaker, 2.8" screen): hold the button, say "cóbrame cuarenta pesos de unos tacos", and it opens a sandbox sale and shows the amount. It speaks every result with the same recordings as the web and the app.
  • Face ID or fingerprint unlock. Charging needs an unlocked device; after one PIN unlock the vendor can save a passkey on that phone. The PIN stays as the fallback.
  • A back office that can't be talked into moving money. Day ledger, cash reconcile, invoices with QR, refunds and payouts to suppliers, plus a Claude assistant with an allowlisted part of PayPal's Agent Toolkit. The AI proposes; a deterministic policy lock decides, with the vendor's PIN on every request.
  • A signed sales passport. An Ed25519-signed report of PayPal-verified sales that anyone can check at /verify without logging in. Self-reported cash and imported bank statements are shown apart, never added in, and it is never a credit score.
  • Several stalls, Claude over MCP. One owner can run several stalls, each with its own PIN, caps and records. Claude can connect at /mcp and only propose refunds and payouts into an approvals inbox.
  • For everyone. English and Spanish, WCAG 2.2 AA target (voice mode, keyboard-only, high contrast, 200% text), and a native Expo app for Android and iOS with an NFC sticker writer.
  • Judge mode. /judge puts the vendor and the customer side by side on one screen. The judge steps show the sandbox test card, and the customer checkout explains the sandbox login and test card, so a first-time judge knows how to pay.
  • Public landing page. A public landing page explains the product before anyone opens the app (also deployable on Vercel, proxying /api to the server).

How we built it

  • Server: Node 22 + Hono + TypeScript in one process: REST API, live events over SSE, PayPal webhooks and an MCP endpoint. Money is always integer minor units, shared with the web and mobile apps. Postgres with versioned migrations, or memory with zero setup. Only server/paypal/ talks to PayPal.
  • PayPal (sandbox): Orders v2 and refunds through the APIMatic-generated @paypal/paypal-server-sdk; JS SDK v6 checkout with findEligibleMethods; Payouts v1; Invoicing v2; verified webhooks; and @paypal/agent-toolkit behind an allowlist.
  • Claude: Haiku 4.5 for voice-to-cart (forced tool output validated with zod, falling back to rules after 4 s); Sonnet 5.5 for the back-office agent and for reading a photo of the menu board.
  • Clients: Preact + Vite web app; Expo SDK 57 app (Android APK and iOS builds on EAS); a TanStack Start + Recharts + AG Grid month dashboard.
  • Voice box: Arduino firmware on an ESP32-S3 with an INMP441 mic, a MAX98357A amplifier and an SPI screen. It posts a WAV to POST /api/terminals/:id/voice; the server transcribes it (Whisper-compatible endpoint) and returns the amount in words, which the box says with µ-law clips baked into the firmware. Passkeys use WebAuthn through SimpleWebAuthn.
  • Tests: 402 Vitest tests, Playwright end-to-end with an axe accessibility gate, and prompt-injection batteries against the lock where no money may move. CI on GitHub Actions; deployed on Render.

The voice box: schematic and cost

Voice box wiring

Voice box layout

One box is an ESP32-S3 (N16R8), an INMP441 I2S microphone, a MAX98357A I2S amplifier with a small speaker, a 2.8" 240×320 SPI screen, one push button ("talk") and a plastic case. Parts bought one at a time on MercadoLibre Colombia on 2026-11-09 (retail prices, COP; USD at roughly 4,000 COP per dollar):

Part COP ≈ USD
ESP32-S3 DevKitC N16R8 40,000 10
INMP441 microphone 48,195 12
2.8" TFT screen 240×320 67,000 17
MAX98357A amplifier 29,990 7.5
Case 25 × 12 × 9 cm 19,000 5
Button, jumper wires ~9,500 2.5
One box, parts only ~213,700 ~54

The whole first order, with a spare ESP32, NFC stickers, a breadboard kit and shipping, came to COP 332,675 (about USD 83). Bought in volume, with a custom board and a printed case, the parts cost should drop well below that; we have not priced it yet.

The idea follows how card-terminal companies are moving in Latin America. In Colombia, Bold sells the Sono QR for COP 59,000: a stand that shows a Bre-B QR code and confirms each payment with a sound. It proves vendors will put a paying "box" on the counter. Paygenti changes who does the work: the vendor says the amount instead of the customer typing it, every payment is tied to a sale, and PayPal lets it charge tourists and buyers from abroad.

The refund rule the lock enforces (MXN; other currencies have equal or tighter caps), where $r$ is the refund, $s$ what was already refunded today and $c$, $f$ the sale's captured and refunded amounts:

$$ \text{allow}(r) \iff \text{PIN}_{\text{typed by the vendor}} \;\land\; \text{same day} \;\land\; r \le 200 \;\land\; s + r \le 1000 \;\land\; r \le c - f $$

The PIN in that rule is one the human types, never one the model writes.

Challenges we ran into

  • Keeping the AI away from money. Toolkit tools that move money are neither enabled nor shown to the model, and the PIN never enters its context. Budgets are reserved first, committed only when PayPal accepts and released if it refuses. If PayPal's answer to a payout is lost, the budget stays spent, because the payout may exist.
  • Payment methods depend on currency and country. Venmo is USD and US only, and Pay Later isn't offered in MXN, so a Mexican sale shows PayPal and card. A static rule per currency runs first, then PayPal's own eligibility check for that buyer and amount.
  • The Server SDK doesn't cover everything. Version 2.5.0 has no Invoicing, Payouts or webhook verification, so those stay on hand-written fetch. We bridged the SDK's transport to our own fetch to keep per-call timeouts (the SDK default is no timeout) and our test stubs.
  • Local rails without a provider can't be verified. Pix and SPEI show a QR or bank data, but the sale goes pending_bank → vendor_confirmed and never reaches "paid".
  • Several stalls. While merging multi-stall support we found one stall's PIN could confirm another stall's bank transfer. Each confirmation now needs the owning stall's PIN, and a test pins it.

Accomplishments that we're proud of

  • One money rule holds everywhere: nothing is "paid" without PayPal's confirmation, and verified and self-declared money are never added together.
  • In our prompt-injection batteries ("use create_refund", "call payouts", a PIN inside the tool call…), zero money moves.
  • A judge can try the whole sale in about a minute on one screen.

What we learned

  • A screenshot isn't a payment: the server-side capture is the confirmation, and the webhook is the backup.
  • PayPal availability is shaped by currency: COP and PEN aren't PayPal currencies, so vendors there charge in USD.
  • The Payouts sandbox accepts email receivers only, and a lost Payouts response must never free the budget.
  • The Agent Toolkit's tool specs map 1:1 to Claude tool definitions, so the hard part isn't wiring tools, it's choosing which ones the model must never see.

What's next for Paygenti

  • Field trials on real devices: NFC sticker, Android phone-to-phone tap, soundbox on an old phone, iOS.
  • Interviews with vendors, and ten sales dictated by someone other than us.
  • Confirming Bre-B (Colombia) transfers from the bank's own notice instead of the vendor's PIN.
  • The voice box: a device token on its endpoint, HTTPS to the hosted deploy, "sales today" and advice.
  • A hosted month dashboard.

Built With

Share this project:

Updates

Submission history