Esper: Reimagining Commerce Through Gesture, Voice, and Trustworthy AI

Inspiration

Online shopping is convenient, but deciding is still hard. A careful purchase usually means five open tabs, a price-history site, and a gut check on reviews that may not be real. Shoppers do not lack options. They lack a clear way to decide and a reason to trust what they see.

We wanted shopping to feel more natural, interactive, and transparent. That idea became Esper: you say what you need, reach into a shopping board with your hands to compare options, and only buy what a transparent scoring model and a safety layer both stand behind. Instead of clicks and search boxes, shopping feels like a conversation with a well-traveled friend who knows products.

What It Does

Esper turns one request into a trusted purchase.

Say what you need. "Get me ready for a camping trip this weekend" becomes a plan of items to buy. Esper asks follow-up questions (where, when, how many people, budget, what you already own) and updates the plan as details change.
Decide with your hands. Real-time hand tracking lets you point, pinch to select, swipe to reveal or dismiss options, and drag one listing onto another to compare them side by side. Mouse and keyboard always work too.
See why a product wins. Every listing gets one score you can read: $$ \text{score} = (\text{bang per buck} \times \text{relevancy}) \times \big(f + (1 - f)\cdot \text{trust}\big) $$

where $f$ is a trust floor you control with a slider. Bang per buck compares the landed cost (price minus coupon plus shipping) with that item's own recent price history, so "cheap" means cheap for this product. The comparison sheet shows every axis, including the ones the winner lost.

Shop safely. Before Esper fetches reviews from a site, links to it, or sends you to checkout, a website safety gate checks the URL, the exact merchant origin, and phishing reputation, and it holds anything uncertain instead of guessing. Review analysis removes duplicates, quotes real reviews as evidence, and flags reviews that try to instruct an AI.
Check out with consent. The assistant can choose an offer, but you review and confirm payment on screen (Stripe test mode or a labelled mock order). Silence is never consent. For camping trips, Esper also grounds its kit in real evidence: NASA weather and terrain data, the forecast, and nearby wildfire observations, and it can explain why an item is on the list ("NASA POWER shows rain on most of these days").

For businesses, shoppers who arrive already decided are easier to convert and less likely to return what they bought, and trust becomes a reason to choose a store.

Esper was developed with the Visa main track in mind, alongside the SpaceXAI, SideShift, and T-Mobile side tracks.

How We Built It

Esper is a full-stack web application served from one origin.

Frontend: Next.js 15 (App Router), React 19, and TypeScript, with MapLibre GL for maps. An earlier Vite and React prototype and a gesture assessment lab are kept in the repo.
Gesture pipeline: our own TypeScript package built on MediaPipe Hand Landmarker. Tracking runs in a Web Worker on the GPU (with a CPU fallback), so video never leaves the browser. A One Euro filter smooths the cursor, and a pure state machine turns landmarks into events like select, drag, drop, swipe, and two-hand expand or collapse. Every threshold lives in one config file with a live tuning panel.
Voice: Grok Voice (xAI) is the shopping companion on the main trip board, and a Gemini Live relay powers the Esper assistant on the shopping page. Both call the same shopping tools (create a plan, open an item, compare listings, pick a listing, set the trust floor, start checkout). ElevenLabs provides speech-to-text tokens and low-latency text-to-speech. API keys never reach the browser.
Backend: an Express gateway (Node) serves the app, connects store APIs, and runs checkout. A Python FastAPI service runs the agent, scoring, review analysis, and website safety.
Retrieval and ranking: products come from store APIs (Best Buy, eBay Browse, Kroger, Etsy, Shopify, WooCommerce, UPCitemdb, Open Food Facts, Icecat, DummyJSON, Fake Store) and marketplace search with Selenium (with fixtures when sites block us). Search fit combines semantic similarity, attribute match, trust, and feedback: $$ \text{fit} = 0.45\,\text{semantic} + 0.25\,\text{attribute} + 0.20\,\text{trust} + 0.10\,\text{feedback} $$

Semantic similarity uses sentence-transformer embeddings when available and hashed character n-gram vectors otherwise. The language model only recommends products the tools returned, and a grounding check catches claims that do not match real listings.

Safety: the website gate (URL rules, exact origin registration, PhishTank reputation, public-only DNS with no redirects), review evidence analysis, prompt-injection checks on tool results, explicit-yes checkout, a circuit breaker, and a kill switch after repeated anomalous turns.
Payments: Stripe Checkout in test mode; the server refuses live keys.
Testing: Vitest, Node's test runner, pytest, and Playwright end-to-end tests, plus a 14-task gesture assessment lab that replays labelled hand landmarks through the real controller and checks the UI result. ## Challenges We Faced

Making gestures feel instant. Early hand tracking lagged, and gesture navigation falls apart without immediate visual feedback. We moved tracking into a Web Worker on the GPU, capped the webcam at 640 × 480 at 30 fps, tuned the One Euro smoothing, and moved the cursor with direct DOM writes instead of React state. That balanced accuracy with responsiveness.

Stopping accidental actions. A hand that is just moving should do nothing. We made actions require deliberate poses (a pinch, a grab, or a two-hand gesture) and built replay tests for cases like losing one hand while the other keeps dragging.

Coordinating gestures, voice, and live recommendations. The voice assistant, the gesture board, and the mouse all edit the same plan. We gave the assistant a tool that reads the current board state, so "compare these two" refers to what is actually on screen, and treated on-screen product text as data, never instructions.

Ranking that people can trust. Generating a confident answer was easy; justifying it was not. We replaced a seven-weight additive score with three named terms multiplied together, so a rock-bottom price cannot hide a weak seller, and we made the trust floor adjustable without refetching anything.

A hostile web. Marketplaces block bots, providers time out, and reviews can carry injected instructions. We added fixtures that run the same formula when scraping fails, and a safety gate that holds instead of assuming a site is safe.

Merging parallel work. Four people built gestures, voice, the backend, and the frontend in parallel, then merged them into one application with one origin and shared contracts.

What We Learned

Technical performance and user experience are the same problem. A gesture can be recognized correctly and still feel wrong if its response is late or unpredictable.
Hands and voice are good at different things. Voice captures intent ("get me ready for camping"); hands are better for choosing ("this tent, not that one").
Transparent ranking builds more trust than a confident answer. Showing the formula, and the axes a winner lost, made results easier to believe.
Safe AI systems fail closed. "I don't know" should mean "stop", not "probably fine".
We gained hands-on experience with real-time computer vision, Web Workers, voice agents with tool calling, retrieval and ranking, and web security.

What's Next for Esper?

Official product APIs and affiliate feeds in place of scraping, plus more review sources.
Embedding-based relevancy at scale and saved trust preferences per account.
Better gesture calibration and measured live-camera accuracy across lighting and devices.
Broader retailer integrations and richer explanations behind each recommendation.
The same spatial interface on headsets and other spatial displays. We believe the future of commerce is more than finding products. It should feel natural, engaging, and trustworthy. With Esper, we have taken our first step toward that future.