Spotted.ai

Spot It. Buy It. Spotted.ai.

Inspiration: B2C Online shopping is a 7.4T dollar industry, and more recently shopping inspiration has increasingly come from the form short videos on platforms like YouTube, Instagram and TikTok. The problem is that finding products featured in those videos can be frustrating. Existing AI assistants often rely on transcripts or video descriptions, which miss products that simply appear on screen.

The idea came from a simple team observation: if someone sees a lamp, couch, skincare product, or outfit from a celeb they love in a video, they should be able to instantly find and buy it. One of our teammates even joked that his mom shops from TikTok all the time and would use this every day, so we wanted to build the missing link between social content and e-commerce.

What it does: Spotted transforms social videos into interactive shopping experiences. Users simply paste a YouTube, TikTok, or Instagram link (or upload a video if retrieval isn't available). Spotted.ai then: Retrieves the video and samples frames every second. Uses OpenAI Vision to identify products appearing on screen. Avoids redundant analysis by recognizing nearly identical scenes. Performs reverse image search to locate potential product listings. Uses OpenAI again to verify that the discovered product actually matches what appeared in the video. Displays timestamped shopping results so users can jump directly to the moment a product appears and open a verified shopping link. Rather than guessing products from captions or transcripts, Spotted analyzes what users actually see.

How we built it: Our workflow centered around rapid iteration with OpenAI. The pipeline:

Retrieve the video from a supported platform.
Sample video frames approximately every second.
Compare consecutive frames and skip nearly identical scenes to reduce unnecessary Vision calls.
Send unique frames to OpenAI Vision to detect requested products.
Crop detected products and perform reverse image search using SERP API.
Uses OpenAI Vision a second time to compare the original object against candidate shopping results and verify the best match.
Return timestamped shopping links inside an interactive interface.
During development we worked collaboratively through Git, each team member iterating on different parts of the pipeline while continuously testing and improving detection accuracy.

Challenges we ran into: The biggest challenge wasn't identifying products, it was identifying the right one. Early versions correctly recognized objects like lamps or pillows but often returned visually similar products instead of exact matches. We solved this by introducing a second AI verification step that compares the detected object with the retrieved shopping candidate before displaying results. Performance was another challenge ; vision models are computationally expensive, so repeatedly analyzing nearly identical frames became inefficient. We optimized the pipeline by detecting similar scenes and only analyzing new visual content. We also encountered networking restrictions during the initial build as it was done during a hackathon, that prevented direct access to some services, requiring us to locally host portions of our application.

Accomplishments that we're proud of: Built an end-to-end AI shopping experience in a single hackathon and continued to iterate.
Successfully identified products directly from video frames rather than relying on transcripts. Implemented scene deduplication to reduce redundant AI processing. Added multi-stage AI verification to improve product accuracy. Delivered a working demo that impressed judges, who commented that it felt different from existing AI shopping experiences because it truly understands visual content.

What we learned: This project reinforced that AI products are as much about system design as they are about model selection. We learned how important iterative prompting, verification pipelines, and optimization techniques are when building reliable multimodal applications. Rather than depending on a single model call, combining OpenAI Vision with the reverse image search and secondary verification dramatically improved result qualities. We also learned that user trust depends on accuracy levels. Returning one verified product is far more valuable than returning ten incorrect ones.

What's next for Spotted.ai: We're excited to continue building Spotted beyond the Open AI Codex hackathon.

Future plans include: Creation of user accounts and shopping lists, navigational through browser extension/app for one-click shopping Personalized recommendations based on saved products and shopping preferences Price tier comparison across multiple retailers and mapping geographics Opportunities to link shopping list directly into user retailer account for faster shopping experience Saved searches and purchase histories Faster inference using optimized and fine-tuned vision models Support for additional inclusion of video platforms and livestreams

Our long-term vision is to make every social video instantly shoppable, allowing users to discover, save, compare, and purchase products directly from the content that inspires them.