Glimpse

Inspiration

Have you ever scrolled on TikTok or Instagram and asked yourself, “I wonder where this creator got that item?” Glimpse was built to answer that exact question. By analyzing what users pay attention to in short-form videos, Glimpse helps surface the products viewers are actually interested in.

What it does

Glimpse turns short-form videos on TikTok and Instagram into a seamless shopping experience. We are redefining the traditional "video -> google search -> product lookup -> purchase" pipeline by reducing the process into just two steps: view and purchase.

How we built it

Glimpse is currently built as a web extension MVP, with the user interface developed in React and TypeScript, and the backend powered by Node.js and Fastify. This architecture allows us to coordinate real-time eye-tracking, product detection, and user profile building seamlessly.

For eye-tracking, we use EyeGestures, an open-source solution that leverages native webcams to accurately determine where a user is looking. When a user focuses on a specific item for a certain duration, Glimpse captures that image and uploads it to Cloudinary for storage and easy access. The image is then analyzed using Google’s Gemini Vision, which identifies the main product in the frame. After this, Cloudinary performs smart cropping using the data provided by Gemini to isolate the product, enhancing clarity and removing unnecessary background clutter.

Once the product image is prepared, it is then sent to SerpApi for reverse image search, which finds matching or similar products online. This data is then returned to the frontend web extension, allowing users to view and interact with the product instantly. At the same time, the captured gaze data and product interactions are sent to Backboard.io, where we build a richer user profile to enable personalized recommendations over time.

For secure hosting, we are currently using Vultur since it is powerful enough to support our eye-tracking software.

Challenges we ran into

We struggled a lot with the eye-tracking model, only figuring it out less than 10 hours before submission. We originally intended on creating our own model using OpenCV, however, we realized this was an unfeasible task. We spent the majority of our time researching different models, learning how these models implement the use of vectors to accurately detect where a user's gaze lands on.

Additionally, many of these models had very poor documentation and were many years old. This made it difficult for us to actually understand how the model was supposed to work.

Accomplishments that we're proud of

We are proud of being able to build Glimpse in the short period. We are especially proud of overcoming the large hurdle of the eye tracking software, allowing Glimpse to actually function. Additionally, we experimented with a lot of new tools such as Cloudinary and Backboard.io.

What we learned

We learned that it is important to adapt and pivot, especially in times of stress. A lot of our struggles came from the fact that we learned the importance of pivoting too late in our Hackathon journey. If we had pivoted earlier, we would most likely have not went through the specific set of challenges that we did.

What's next for Glimpse

One of the biggest things for Glimpse is actually integrating the software within TikTok or Instagram. Since Glimpse is currently an MVP, we were unable to actually do so. Our whole vision of the app is to integrate Glimpse fully within these social media platforms, increasing our already high conversion factor.

Glimpse also has another huge potential in growing in the B2B2C and B2B space. For example, instead of using eye-tracking to recommend products, we can also provide services for businesses to use eye-tracking data to analyze ad performance.