VoiceCart: Screen-Free Shopping Agent

A voice-only shopping agent for blind users: search many stores, hear 3 clear options, confirm by spoken passphrase and pay with PayPal. English, Hindi, Tamil. No screen needed.

Comment

Inspiration

At least 2.2 billion people live with a vision impairment, and India alone has about 5 million blind and roughly 70 million vision‑impaired people. Online shopping still assumes eyes: product photos without alt text, cluttered listing pages, and checkout forms that screen readers trip over. Voice assistants help a little, but each one is locked to a single retailer and still hands you back to a screen the moment you try to pay.

We asked a simple question: what if the entire journey — from "I need running shoes" to "here's your receipt" — never needed a screen at all?

What it does

Saathi (Hindi for "companion") is a voice‑only shopping agent built around how blind users actually shop.

  • Speak naturally in English, Hindi or Tamil, or mix them mid‑sentence.
  • Cross‑merchant search across 100M+ products from thousands of retailers using Channel3 — not one store's catalogue.
  • Exactly three options, read back with what matters when you can't see: price, merchant, stock, return policy and a precise non‑visual description ("charcoal grey, not black").
  • Interrupt any time — "wait, tell me more about the second one" works mid‑sentence.
  • Statement‑style confirmation: Saathi states the item, merchant and total out loud before asking for anything.
  • Spoken‑passphrase payment gate: only your passphrase plus the word "confirm" can trigger a charge. Ambiguous answers never pay.
  • PayPal checkout with no redirect: after a one‑time approval at onboarding, every purchase is charged to your saved PayPal wallet and the receipt is read back to you.
  • "Cancel that" within two minutes triggers a PayPal refund. "What did I buy?" reads your PayPal transaction history.
  • Adjustable speech rate and audio earcons for searching, results, success and error — so you're never left in silence.

How we built it

  • Voice: OpenAI Agents SDK with the Realtime API (gpt-realtime-2) for full‑duplex speech‑to‑speech, barge‑in and tool calling, over WebRTC. Sarvam AI Bulbul/Saaras as the Indic speech fallback for natural Tamil and Hindi output.
  • Payments: PayPal Agent Toolkit (@paypal/agent-toolkit) for get_order, create_refund and list_transactions; PayPal Orders v2 API with Payment Method Tokens v3 (Vault) — the user approves a billing agreement once, then every purchase is created with payment_source.paypal.vault_id and captured without a browser redirect. Idempotent charges via PayPal-Request-Id. All in the PayPal sandbox.
  • Products: Channel3 POST /v1/search with price and availability filters, and GET /v1/lookup for details, returning multi‑retailer offers in one call.
  • Safety: a deterministic, server‑side confirmation state machine (quote → spoken total → passphrase match → "confirm" → spend cap → charge). The LLM proposes; only the user's voice authorises. The model never generates a price — every number is read from Channel3 data.
  • Stack: TypeScript / Node.js, deployed on Render. A minimal WCAG 2.2 AA fallback page with an aria-live transcript for sighted helpers and judges.

Challenges we ran into

  • Checkout wants a screen. Standard PayPal checkout requires a browser approval on every order. We redesigned payment around a one‑time vaulted approval plus a spoken passphrase gate so purchases are genuinely screen‑free afterwards.
  • Product feeds don't carry what blind users ask first. Delivery time and return policy aren't in most catalogue data, so Saathi says "not provided by this merchant" rather than letting the model guess.
  • Making three options comparable by ear. It took many iterations to find a reading format — name, price, merchant, stock, returns, then details on request — that users could hold in their head.
  • Code‑mixed speech. Hindi‑English sentences like "mujhe black running shoes chahiye, eighty dollar ke andar" broke early prompts until we treated language switching as the default, not an edge case.

Accomplishments that we're proud of

  • A complete purchase, from spoken request to PayPal receipt, with the laptop lid closed and the phone face‑down.
  • An agentic payment flow where the model can never pay without the user's explicit voice confirmation.
  • Real conversations in English, Hindi and Tamil, including mid‑sentence interruptions.
  • Post‑purchase servicing by voice: refunds and purchase history through the PayPal Agent Toolkit.

What we learned

Blind shoppers don't want more information — they want less, in the right order. Published research with blind users found three options is the practical limit and that confirmations should be statements, not questions. Trust at the payment step comes from the agent never inventing a price and never acting without being told to.

What's next for Saathi

  • Indian merchants and INR payments as catalogue and payment coverage allow.
  • Routing payments directly to merchants via PayPal Agent Ready and Store Sync.
  • Phone‑call access (SIP) for users without smartphones.
  • Co‑design sessions with blind users' organisations in Bengaluru, and a public accessibility report.

Built With

Share this project:

Updates

Submission history