toes

Inspiration

Online shopping is fragmented by design: a customer browses on one screen, compares specs on another tab, reads reviews somewhere else, then gets redirected through a multi-step checkout just to complete a purchase. Meanwhile, most of the merchants who'd benefit most from an AI-native shopping experience, small and mid-sized businesses, don't have the engineering resources to build one. The Visa challenge asked us to imagine what commerce looks like when discovery, decision, and payment collapse into a single conversation. We wanted to prototype that seriously: not a chatbot bolted onto an existing storefront, but an agent that could actually reason over a real catalog and trigger a real (if mocked) payment, with the trust and consent safeguards that kind of agent-initiated transaction demands.

What it does

Toes lets a merchant upload their product catalog with no code and no integration work, and instantly stand up a category-trained AI shopping agent for their customers. On the customer side, a chat interface handles the entire shopping journey: the agent answers product questions, compares options, builds a cart, and walks the customer through checkout, all inside the same conversation. Before any money moves, the customer sees an explicit transaction preview and must confirm through a distinct authorization step. Payment only fires after that consent is captured, and every transaction is logged into a full audit trail. The result is a complete discover, decide, and pay flow that never leaves the chat, backed by a payment layer modeled on Visa's payment stack.

How we built it

We split the system into two parallel tracks, a merchant-facing site and a customer-facing site, each with one FastAPI + uvicorn backend. Merchants upload a CSV catalog, which we embed locally using Ollama's nomic-embed-text model for semantic search, with GPT-5 assisting in normalizing messy or inconsistent product data on ingestion. On the customer side, a GPT-5-powered conversational agent uses that embedded catalog to answer questions, generate comparisons, and build a cart through structured tool calls rather than free-form text. The payment layer is its own FastAPI service backed by asyncpg, deliberately structured as three separate steps (preview, authorize, confirm) instead of one collapsed "checkout" call, so that consent is a first-class, auditable part of the flow rather than an implementation detail. Visa's actual payment stack is mocked for this prototype, standing in for the tokenization and authorization layer a real integration would call.

Challenges we ran into

The biggest challenge was resisting the obvious, convergent solution. Early on, our plan looked identical to what every other team would likely build: a catalog-aware chatbot with a payment button. So we spent real time identifying where the actual hard problem lived: the trust and consent gap at the moment of payment, not the chat UX around it. Deciding how far to take "no-code" merchant integration was another open question. We ultimately scoped down from a full embeddable widget with script-tag deployment to a standalone web app, so we could spend our limited time on the AI reasoning and payment consent flow instead of cross-platform embedding, and out of respect for platforms like Shopify actively restricting third-party checkout scripts. We also had to be deliberate about splitting user-side authorization (proving it's really the customer) from payment-side authorization (the actual charge). It would have been easy to merge these into one step, but that separation is core to the trust story we wanted to tell.

Accomplishments that we're proud of

We're proud that our payment flow treats consent as a structural guarantee, not a suggestion. The agent is architecturally incapable of skipping the preview or confirmation step before charging a card, even a mocked one. We're also proud of the catalog ingestion pipeline: turning an arbitrary merchant's messy CSV into a normalized, embedded, searchable product catalog with no manual setup is exactly the "no-code" promise the challenge asked for, and it works end-to-end. Getting a real, working demo, not just slides, covering discovery, comparison, cart building, and a full consent-gated mock payment, all inside one chat thread, in the time we had, is something we're genuinely happy with.

What we learned

We learned a lot about the layers underneath a single "Pay Now" button: tokenization, authorization, settlement, fraud and dispute handling, and how much of that complexity Visa's actual payment stack absorbs on a merchant's behalf. We also learned that an "AI agent" is only as trustworthy as the guardrails around it. The interesting design problem in agentic commerce isn't getting an LLM to recommend a product, it's making sure it can never move money without a human clearly and explicitly saying yes. On the engineering side, we learned the practical tradeoffs between local and hosted models. Running embeddings locally on Ollama while keeping conversational inference on a hosted model turned out to be a meaningful cost and latency decision, not just an implementation detail.

What's next for toes

• Swap our mocked payment flow for a real Visa Developer Platform sandbox integration
• Expand beyond a single product category with more category-specific comparison logic (sizing, compatibility, dietary filters, etc.)
• Bring back the embeddable widget/script-tag approach so the agent can live directly on a merchant's existing storefront
• Replace mock authorization with real identity verification (e.g. passkeys or SMS OTP)
• Add a merchant-facing analytics dashboard showing what customers ask, where they drop off, and conversion by category
• Explore voice as an input mode, and cross-merchant discovery for a true "shop a category, not just a store" experience

Built With

• css
• fastapi
• html
• javascript
• python