Inspiration
Online shopping is easy if you're confident with technology. For many older citizens, it's hard work: lots of websites, product details full of jargon, delivery fees that show up at the end, and no easy way to tell a reliable seller from a risky one. Many people end up asking a family member for help, or not shopping online at all. We kept coming back to one question: what if you could shop online the way you'd shop in a good local store? You'd just tell someone what you need, they'd find the right thing, point out anything dodgy, and only sell it to you once you say yes. My dad is 74 and lives on his own, roughly 4,500 miles away from me. There's no one readily available to help him with online shopping, as I can't be there in person. It got me thinking: what if he had a team of AI agents that could help him shop safely, simply by speaking in natural language?
AMISA came from that idea. It's an AI shopping assistant you can simply talk to, and it does the research and the checking while you stay in control.
What it does
AMISA lets you shop by having a conversation, by voice or by typing.
- Just say what you need. Tap the orb and say "I'm looking for a toaster". AMISA asks one or two simple questions (your budget, then what matters and when you need it), the way a helpful shop assistant would.
- AI agents do the research. A main agent hands work to specialist agents (Research, Safety, Order and Delivery). They search five demo stores, compare the full delivered price, delivery time and returns, and recommend the best overall option, not just the cheapest.
- It protects you, with evidence. AMISA flags listings with warning signs: headlines that contradict the specifications, prices far below similar items, huge "was" prices it can't verify, vague returns, very slow delivery and new sellers with no contact details. It shows exactly what it found and what it couldn't check. It never calls a seller a scam, and it never claims anything is guaranteed safe.
- Nothing is bought without your approval. AMISA reads back the exact item, store and total. Only a clear "I approve", or a press of the Approve button, moves things on. You then pay on PayPal's own secure page; AMISA never sees your PayPal login. If you prefer, you can save a PayPal account once, on PayPal's own site, so that approving pays straight away.
- It keeps helping after checkout. You can ask "Where is my reading lamp?" or "Why is my toaster delayed?", and the answers come from stored tracking events.
- Family and carers, if you want them. You can invite a family member or carer, even by voice, to follow your orders. You choose whether they can place orders, and they never see your conversations or payment details.
- The whole app works by voice or by click. You can say "take me to my orders", "change it to light mode", "start a new conversation" or "bye". You can interrupt AMISA by speaking, and it remembers your past conversations.
How we built it
- Web app: Next.js and TypeScript, with a calm black-and-white interface designed for readability (large text option, high contrast, keyboard support). Accounts, orders,
conversations and each session's working state are stored in Postgres on Neon.
- Voice: Deepgram Flux for live speech recognition, which detects when you've finished speaking, so pauses and "um"s don't cut you off. Deepgram also provides AMISA's spoken voice. A small WebSocket relay on our own server keeps the Deepgram key off the browser, and speech starts playing while it's still downloading.
- The brain: a main agent on Gemini (via OpenRouter) that calls specialist agents, each with its own instructions and only the tools it needs.
- The AI decides, code enforces: searching, ranking, the recommendation, trust findings, prices, totals and payment details are all ordinary code the AI can't change. No agent has a tool that can approve or pay. Every reply passes through a guard that blocks invented prices, "this seller is a scam", "guaranteed safe" and pressure tactics. If the AI fails, fixed rules answer instead, and the customer is told.
- Payments: PayPal Sandbox Orders v2. The server re-checks the total, price, stock and permissions before creating and capturing the order. Saved PayPal uses PayPal's vault, so approving pays straight away.
- Demo stores: five fictional stores, each with its own data format behind a shared adapter interface. One store has deliberate warning signs, so the trust checks can be demonstrated honestly.
- Testing: almost 300 automated tests. They cover checkout approval rules, budget filtering, trust findings, the agents (using a scripted fake model, so no network calls) and voice commands. We also ran full conversations in a browser using a simulated microphone.
Challenges we ran into
- Natural turn-taking. Early versions cut people off mid-sentence or waited awkwardly long to reply. Switching to Deepgram Flux's end-of-turn detection, plus a fallback when room noise keeps it waiting, made conversations feel natural.
- Speed. Waiting for an AI model just to ask "what's your budget?" felt slow. Simple questions now come instantly from fixed rules, the agents do the heavy work, and speech starts playing in about a second instead of waiting for whole sentences to download.
- Safe spoken approval. Letting people buy by voice without accidental purchases took care. A bare "yes" never buys anything. AMISA reads the order back first, the server checks the spoken words again, and anything unclear leaves the decision on screen.
- Deploying to serverless. On Vercel, one request can reach a different server instance from the last, so order reviews and conversations kept "disappearing". We moved all working state into Postgres and load it fresh on every request.
- Interruptions. We wanted people to be able to talk over AMISA, without AMISA mistaking its own voice for theirs, and without losing the first words of an interruption.
- Pronunciation. Getting the speech voice to say "AMISA" correctly (a-MEE-sa) took a spelling workaround.
Accomplishments that we're proud of
- A complete journey you can do entirely by voice: from "I'm looking for a toaster" to a real PayPal Sandbox checkout, then a delivery question afterwards.
- Trust warnings that show their evidence and are honest about what AMISA can't know, rather than a vague "scam score".
- A clear safety design: the AI chooses what to do, but it can never touch prices or payments, and it can't buy anything without the customer's explicit approval.
- An interface designed for older people without looking childish or medical, which anyone can use by voice or by clicking.
- A demo that keeps working if the AI service goes down: fixed rules take over and say so.
What we learned
- For older users, trust comes from transparency. "Here's what I checked and what I couldn't" lands far better than a confident yes or no.
- Voice is about timing as much as words. Small delays and interruptions decide whether a conversation feels natural.
- Agentic AI works best with firm boundaries. Letting the model plan while plain code handles money and rules made AMISA both capable and safe.
- Serverless hosting changes how state must be handled. Anything that matters has to live in the database, not in a server's memory.
What's next for AMISA - Agentic Multimodal Intelligent Shopping Assistant
- Real stores: connecting to real retailer catalogues and APIs, with the same transparent trust checks.
- Family Guardian alerts: an optional notice to a trusted family member when a purchase is over a set budget or has serious warning signs, always with the shopper's consent.
- Production payments: live PayPal with fraud-prevention data, and more payment options.
- More ways in: text messages for family invitations, more languages, and phone-call access for people who don't use apps.
- Real delivery tracking from couriers, replacing the simulated tracking.
- Testing with older shoppers to refine the conversation, the wording and the accessibility features.
Built With
- css
- deepgram
- deepgram-flux
- gemini
- html
- javascript
- multi-agent
- neon
- next.js
- node.js
- openrouter
- paypal
- paypal-orders-api
- paypal-sandbox
- paypal-vault
- postgresql
- react
- speech-to-text
- sql
- text-to-speech
- typescript
- vercel
- web-audio-api
- websockets
- zod