Shopi

Never run out again! Shopi sees when essentials run low and auto-restocks before you notice. No lists. No errands. Effortless, invisible shopping. Your world, always stocked.

Comment

Inspiration

Repeatedly running out of items and forgetting to reorder? Tired of shopping lists and having to go order every single thing you need? Shopi uses Snap Spectacles and real-time computer vision to monitor stock levels and auto reorder the item via voice or gesture.

What it does

  • Detects containers with low stock and surfaces replenishment prompts
  • Processes images with Gemini via snap-and-buy
  • Matches detected items to an on-device catalog
  • Supports hand-tracking, voice, and pinch inputs
  • Uses AR overlays and TTS for hands-free interaction
  • Instantly prompts the reorder of items

How we built it

Frontend (Snap Spectacles):

  • Lens Studio with TypeScript/JavaScript
  • Spectacles Interaction Kit v0.10.0 for hand gestures
  • HandVisual for 3D tracking and visual feedback
  • Voice transcription using Snap OS

Backend (FastAPI):

  • Python 3+ and AsyncIO
  • ngrok for server deployment
  • Pydantic for schema validation
  • Gemini API (2.5 Flash) for product detection and attribute identification
  • Selenium for Shopify checkout automation
  • Google Search API for product discovery and label-matching

Challenges we ran into

  • Spectacles hand tracking: fixed by enabling disabled HandVisual components in the scene hierarchy
  • FastUI integration: unitPlaneMesh errors were addressed via correct component wiring
  • Image encoding: large camera frames required base64 JPEG and browser telemetry
  • Product matching: refined Gemini prompts to reduce false positives
  • Selenium automation: anti-bot mitigations required randomized input timing

Accomplishments that we're proud of

  • Working AR glasses integration with real-time vision
  • End-to-end path: image → detection → catalog match → purchase
  • Hands-free interaction via gestures
  • Sub-second AI responses with caching
  • Modular FastAPI design enabling future commerce integrations
  • Stable build across TS/JS in Lens Studio and FastAPI backends

What we learned

  • Cross-platform performance requires hardware-aware optimization
  • AR UI benefits from gesture-driven controls and minimal overlays
  • Efficient vision-processing pipelines reduce latency
  • Error handling and graceful degradation are important
  • Fast iteration wins with cloud AI; model fine-tuning remains niche
  • Generative AI prompts need specificity to avoid bias and hallucination

What's next for Shopi

  • Integrate voice and real-time object detection without need for taking a snapshot

Built With

Share this project:

Updates