EyeSearch

Block Diagram

Inspiration

A personalized experience is important for some customers, such as vegetarians, diabetes. Having glasses that integrated with Wegman's shopping experience would be beneficial for those people. Checking ingredients for each item can be cumbersome, therefore, having glasses that can scan all the items that user checks during shopping and give warning and feedback is a deal-breaker for some customers. Our objective is that we wanted to have a system that allows these users to have better customer experience with Wegmans.

What it does

The system takes advantage of pupil glasses by extracting QR and barcode to collect product ingredients. By using this information, provide feedback to the customer while they are shopping.

How we built it

The system is consists of two subsystems, the first system is streaming world camera frames from the pupil headset. In addition to the frame, gaze information such as fixation was collected as well. Then using both information, the fixated location of the frame was cropped and QR and Barcode detection applied.

In order for the system work, first, it needs to scan a key which can be a customer shopper card, in this case, we used a QR code as a proof of concept. After customer scans their id card, then the system links to their account and ready to provide feedback for the customer. The system tracks customer's gaze and only apply detection when the customer focuses on a product.

When the system detects, a fixation, the system looks for QR or Barcode around the fixation location and sends it to the customer's smartphone application which is the second system. Where he/she can see it more detail on a bigger screen. The customer also can add the product in their card from their phone screen, or discard the feedback.