-
-
Look at something and say "buy me this". Pagent grabs the frame and starts working.
-
Not sure? It asks one specific question instead of guessing: "Is it the Pro or the Pro Max?"
-
One confirm card: exact product, retailer, lowest in-stock price. Nothing is paid until you say "yes".
-
The receipt: order ID, amount and status COMPLETED, verified with PayPal. Sandbox payment.
-
The agent at work: Claude looks at the photo, searches Channel3 and compares results. Every step is shown live.
-
After a spoken "yes": the PayPal sandbox order is created, approved by the buyer and captured.
Inspiration
Buying one thing online still means tabs, carts, account forms and a card number. But the moment you actually want something is usually when it is right in front of you: a friend's water bottle, the mouse on someone's desk. I wanted that moment to be the whole checkout. Look at it, say "buy me this", done.
Pagent is a web app that does exactly that, from any phone or laptop camera.
What it does
Pagent is a shopping agent you talk to.
- Point your phone's camera at a real object and say "buy me this".
- Pagent works out exactly what it is: brand, model, size.
- It searches real retailers and keeps only exact matches that are in stock.
- It shows and reads out one card: photo, name, retailer, price.
- You say "yes". Only then does it pay, with PayPal.
- You get a receipt with the order ID and status.
If it is not sure (two sizes, two colours), it asks you a follow-up question instead of guessing. The first full run bought a G PRO 2 mouse for $119.99 from a photo of the one on my desk.
How I built it
- The brain: Claude runs the loop with four tools: search by photo, search by text, propose a purchase, ask the user. There is no "pay" tool. Claude can only name a product.
- The eyes: Channel3's image and text search turns a camera frame into live offers with price, stock and retailer. The server also downloads the result photos and shows them to Claude, so it compares the real picture and not just a title.
- The price: plain code picks the offer, never the model: in stock, lowest price.
- The money: the PayPal Agent Toolkit over MCP, with only create, get and capture order switched on, in the PayPal sandbox.
- The voice and ears: Kokoro for speech and Whisper for listening, both open source and running on my own machine. No cloud speech service.
- The app: TypeScript and Node, a phone-first web page using the back camera, reached over an HTTPS tunnel behind an access code. A live timeline shows every step the agent takes.
The one rule: nothing is paid until you say yes
This rule lives in code, not in a prompt. Every PayPal call needs a ConfirmedPurchase object, and the only way to make one is confirm() after an explicit spoken yes. The amount is frozen from the offer when the card is shown, then checked against PayPal before and after capture. A model that gets confused, or gets tricked by text on a product label, still cannot spend money.
Challenges I ran into
- Smart glasses, tried and parked. I also built an iPhone bridge to stream live video from Ray-Ban Meta glasses. It installed, but live streaming needs an entitlement that only a paid Apple developer team gets. So Pagent is a phone-first web app: nothing to install, and it works with any phone camera.
- The agent could confirm its own purchase. The page reads the card aloud, and the microphone was listening. If Pagent ever said "yes", it could hear itself and pay. Now spoken lines can never contain that word (enforced in code) and the mic is ignored while Pagent talks.
- Matching the exact product. With titles alone, Claude refused good matches because listings did not name a colour. With result photos it matched a Stanley 40 oz in about 7 seconds. The retailer's image host blocked Claude's fetcher, so the server fetches the photos itself.
- Errors that look like success. The PayPal MCP returns API errors as normal results. One wrapper now turns them into real errors, so a failed payment can never look like a paid one.
- A black frame once became a Ray-Ban offer. A fallback cache was too eager. Now empty frames are rejected before the agent starts.
Accomplishments that I'm proud of
- A real end-to-end purchase from a camera frame: order created, approved, captured and verified as COMPLETED in the PayPal sandbox.
- The safety rules are tested, not promised: 104 automated tests cover the purchase gate, money handling, voice commands and the agent loop.
- It was built solo in two days, and it has its own voice.
What I learned
- Keep the model away from the money. Let it see and decide what the thing is, and let boring code choose the price and pay.
- Voice interfaces have feedback loops that screens do not. Anything the app says, it might also hear.
- A vision model matches products far better when it can look at the candidates.
- Pick the hardware last. The agent is the product, and a web page with a camera reaches everyone.
What's next for Pagent
- Smart glasses, with the card spoken in your ear.
- Real retailer checkout. Today payment runs in the PayPal sandbox and retailer fulfilment is simulated.
- Refunds, order tracking and a spending limit you set once.
Built With
- anthropic
- channel3
- claude
- cloudflare
- kokoro
- mcp
- node.js
- paypal
- paypal-agent-toolkit
- typescript
- vitest
- whisper
- zod