Omni

Inspiration

We spend way too much time shopping and researching. Every time we pick up a product in a store, we wonder: "Is this a good deal? Can we get it cheaper online?"
We wanted a tool that could instantly tell us if we should buy now or wait—without having to pull out our phones to manually search.

What it does

ShoppingLens is a real-time AI shopping assistant. Point your camera at any product and it instantly:

• Detects the product using computer vision (Gemini Vision)
• Searches major retailers (Amazon, Best Buy, Walmart, Target, etc.) for current prices
• Compares prices and shows you the best deals in a sleek heads-up display
• Gives you a verdict — Buy or Wait — based on price analysis
• Supports voice interaction via LiveKit for hands-free questions about the product

How we built it

• Frontend: React + TypeScript with Tailwind CSS, featuring a glassmorphism HUD overlay on live camera feed
• Backend: Express.js with WebSocket for real-time communication
• Product Detection: Gemini Vision API to identify products from camera frames
• Price Research: Gemini with Google Search grounding to fetch live prices from retailers
• Voice: LiveKit for real-time speech-to-text and AI responses

Challenges we ran into

• Getting accurate product detection from camera frames—we had to carefully tune prompts to read visible text (brand names, model numbers) rather than just describing "a person holding something"
• Filtering out garbage search results—Google's grounding API sometimes returns internal redirect URLs with no real prices
• Linux audio drivers—we spent way too long debugging Bluetooth microphone issues on Ubuntu

Accomplishments that we're proud of

• The entire detection-to-price pipeline works in under 5 seconds
• Clean, futuristic UI that feels like something from a sci-fi movie
• Real price data from actual retailers, not mock data
• Automatic scanning—just hold up a product and it starts working

What we learned

• Juggling cognitive priorities between vocal and visual result
• Gemini's grounding/search capabilities are powerful but require careful result filtering
• WebSockets + React hooks make real-time UX surprisingly smooth
• Building for camera input requires thinking about frame rates, debouncing, and user feedback
• Always have a backup mic for hackathons

What's next for Omni

• Browser extension—compare prices while shopping online
• Purchase history tracking—know if something you're looking at has dropped in price
• Barcode/QR scanning—faster product identification
• AR overlays—price tags floating next to products in real space
• Multi-product comparison—scan multiple items and compare them side by side

Built With

• gcp
• langchain
• livekit
• overshoot
• python
• vertex