Storefront

The app is configurable and supports many categories, as well as Meta Glasses.
The mobile app shows all the objects.
The browser viewer displays on a larger screen.

Inspiration

You spot a great lamp at a friend's place or a jacket on the bus, and have no idea what it is or what it costs. Reverse image search works one object at a time. We built Storefront for the Visa track at DubHacks 2026 around a simple idea: put a price tag on everything you see.

What it does

Point your phone or Meta glasses at a room, pick a category, and Storefront overlays live labels and prices on every matching object. An agent compares similar products, finds cheaper listings, and saves items for later. An optional voice prompt like "something like that couch, under $500" refines the search.

How we built it

Segment every object matching the chosen category with SAM 3, prompted by the category text.
Track each object across frames so it keeps one identity and one tag.
Label each tracked object with Claude Haiku 5.5 to get a searchable description.
Query Amazon, Google Shopping for matching products and prices.
Agentic layer on top of the listings: compares options, finds cheaper alternatives, refines with voice prompts, and saves items.

Frontend: SwiftUI. Backend: PyTorch, Python, LiveKit.

Challenges

Live inference

Latency budget: a tag that shows up 3 seconds late is useless. Segmentation, labeling and search together blew past real-time, so we split the pipeline: SAM 3 and tracking run on the stream, while labeling and listings calls run once per tracked object and are cached.
Flicker and identity: SAM 3 re-segments objects slightly differently each frame. Tracking keeps one ID per object, so labels stay put and each object is labeled and queried once instead of every frame.
Bandwidth from glasses: streaming full-resolution frames was too slow, so we downscaled.
Exact matches: "gray couch" returns thousands of results; finding the same item took heavy prompt iteration.

Security and privacy

Bystanders: the camera sees people, faces and screens, not just products. We never store the frames.
Untrusted listings: the cheapest result is often a scam or a fake store. We filter by seller and flag suspicious prices.
Admission control: we run the first frame through a VLM to ensure sessions start in a safe, public environment
API keys: keys for the model and listings services live server-side, not in the app.

What we learned

Errors compound across chained models, so ranking matters as much as perception. Real-time is a design constraint, not an optimization. Anything the camera sees is untrusted input. And in AR, less on screen is more.

What's next

On-device segmentation for glasses to cut latency and keep frames private, price alerts on saved items, and secure checkout directly from the overlay.