-
-
Paying for real: confirm, a typed PIN, then a 10-second undo window before PayPal captures.
-
The AI suggests. The shopper decides.
-
Is this my package? Gemini vision compares a delivery photo with the order: 'photo suggests this may not match'.
-
Scam warning: a gift-card request after a scam phrase needs the trusted contact's approval.
-
The Readback: every number comes from PayPal's own order, read aloud before any money moves.
-
PayPal's own record of the three demo orders: one captured, two voided (nothing charged).
-
The trusted contact sees why the order was flagged, and declines; PayPal releases the hold.
A working prototype on PayPal's sandbox. It has not yet been tested by blind or low-vision shoppers; that is the next step.
Inspiration
Blind and low-vision shoppers can often find a product online, then lose it at checkout: pop-ups, unlabelled buttons and payment iframes a screen reader can't get into. About 43 million people worldwide are blind and 295 million more have moderate or severe vision impairment (GBD 2019 Blindness and Vision Impairment Collaborators, The Lancet Global Health, 2021). AI agents can now click through websites for people, but an agent that moves money on your behalf has to be trustworthy, not just capable. We wanted the agent to do the clicking while the person makes every decision. (The demo runs on one fictional store in PayPal's sandbox.)
What it does
Sightline is a voice- and keyboard-first shopping assistant built on PayPal.
In the video: search, the Readback, confirm + PIN, Undo (PayPal releases the hold), paying for real, a scam warning and the trusted contact's decision, the package photo check, and PayPal's own API record of all three orders. Also built, not in the video: the browser extension prototype, listing-vs-photo descriptions, reviews, compare, sizes, allergies, refunds and dispute help, Pay in 4 information, price-drop alerts and purchase history (all in the repo, with tests).
- Find and understand products: at most three results at a time, spoken briefly; descriptions separate what the listing says from what the photo suggests, and say "not stated" instead of guessing. Review summaries, comparisons, sizes and colours.
- The Readback: at checkout PayPal holds the money and Sightline reads back PayPal's own amounts (price, shipping, tax, total) plus the item, size, and the store's delivery date and seller. Nothing is paid until the shopper says "confirm" and types their Sightline PIN (set in Sightline, not their PayPal password); PayPal then captures after a 10-second undo window (undo, and PayPal voids the hold).
- Safety by code, not by prompt: the AI never sees prices (it picks products; code adds the prices it speaks) and can't move money; spending limits and the PIN change only in Settings; allergies are a hard filter; pre-ticked add-ons, subscriptions and "only 2 left" pressure are called out in plain words; a 10-second undo after the PIN; an optional trusted contact who must approve orders over an amount the shopper sets (the prototype makes an approval link; the shopper shares it); refunds behind the same Readback + PIN.
- Scam warnings: gift-card requests and classic pressure lines ("a caller says...", "keep it secret", "pay a fine") get a calm warning. If the shopper has turned on the scam hold in Settings (off by default), their next order (within 24 hours) then needs the trusted contact's approval, who sees why it was flagged; otherwise the usual Readback, confirm and PIN apply. The warning itself says which applies. Keyword rules, so it can miss or over-flag.
- After the purchase, by voice: "is this my package?" compares a delivery photo with the order (Gemini vision, always worded "photo suggests"); "it arrived broken" explains PayPal Purchase Protection and starts a refund from the demo store (not a PayPal dispute); "what did I buy this week"; price-drop alerts on saved items, said once.
- PayPal Pay in 4, information only: when the total qualifies, the Readback mentions "4 payments of X" and "pay later" explains it. The prototype always pays in full; Sightline never picks credit for the shopper.
- Browser extension prototype: brings the same Readback + confirm + PIN gate to other checkout pages (so far tested only on local look-alike pages): it reads the page's own total and payee aloud and holds the pay button until the PIN. No network calls; the PIN is stored only as a hash.
- Works with the tools people already use: replies go to the user's own screen reader (NVDA, JAWS, VoiceOver) or an optional built-in voice, push-to-talk or typing, modifier-only shortcuts, large text and simple modes.
How we built it
- PayPal (sandbox): Vault v3 saves the buyer's PayPal once, on PayPal's own consent page in a full-page redirect (no popup). Each purchase is an
Orders v2
AUTHORIZEwith the savedvault_id; Sightline reads the Readback from PayPal'samount.breakdown, then captures on confirm + PIN or voids on cancel/undo. OnePayPal-Request-Idper checkout attempt makes a repeated "confirm" safe. Refunds use the Payments v2 refund endpoint. - AI in three lanes for speed: fixed commands and the whole checkout run with no AI (instant, deterministic); free-form requests go to a fast Gemini Flash-Lite model with tools (~0.5–1.5 s); heavy work runs offline — Gemini with vision writes accessible descriptions from product photos, which the developer reviewed by hand.
- Stack: Python, FastAPI, plain semantic HTML and JavaScript (role="log" transcript, aria-live, Web Speech push-to-talk), SQLite for order history, 180+ tests plus end-to-end browser tests of the extension, automated axe checks. Built in 16 days, at zero cost.
Challenges we ran into
- PayPal's checkout normally needs a popup; we found that saving the wallet through a full-page redirect keeps PayPal's own login in charge of security while every later purchase stays voice-driven.
- Keeping AI out of the money path: the model kept "helpfully" restating prices, so prices never reach it at all.
- An early test returned the peanut trail mix for a "nut allergy" request, because the search matched the word "nut". Allergies became a hard filter in code.
- Making replies speak exactly once with a screen reader, and running on free-tier AI limits.
Accomplishments that we're proud of
A complete purchase, refund and approval flow designed for screen readers and the keyboard, with every spoken amount traceable to PayPal's data; and safety rules that hold even if the AI misbehaves.
What we learned
Most AI accessibility tools describe things. The hard, unsolved part is AI that acts safely: reading back what will happen and waiting for the person to decide.
What's next for Sightline
Testing with blind and low-vision shoppers (this prototype has not yet been tested by them), real merchant catalogues, taking the extension from look-alike test pages to PayPal's live checkout, and a version for older adults focused on scam protection.
Honest limits
- Sandbox only, fake money. The shop and its reviews are fictional demo data; photos are Wikimedia Commons (credited).
- The demo video's voices are synthetic; the shopper's speech is typed by a script and narrated.
- Pay Later is read aloud but not used: the vaulted checkout pays in full. Dispute help refunds through the demo store; it does not open a PayPal dispute. The scam guard is keyword rules, so it can miss or over-flag. The extension has been tested on local look-alike pages, not on live PayPal checkout.
Links
- Code (MIT): https://github.com/Thomas-Wood-Eng/sightline
- Demo video: https://www.youtube.com/watch?v=IlVZFV1RJDY
Built With
- aria
- fastapi
- gemini
- javascript
- paypal-rest-api
- paypal-vault
- playwright
- python
- sqlite
- web-speech-api