Shelf Aware

Inspiration

My brother has diabetes. When he shops for groceries, he spends extra time reading nutrition labels, but claims like “sugar-free” and “no added sugar” can make the decision more confusing. Serving sizes vary, similar packages contain different ingredients, and the amount he actually eats matters. We wanted to make health-conscious grocery shopping more intuitive and personal, while making it more seamless and less manual. Hold a product, ask a question, and understand the trade-offs without interrupting the shopping trip. This became Shelf Aware: self-aware, intentional grocery shopping.

What It Does

Shelf Aware is an audio-first grocery assistant. It shares the shopper’s visual context and helps interpret the product they are holding through natural conversation. Its core experience brings together product recognition, nutrition-label extraction, user-reviewed preferences, and portion-aware comparisons. The conversation starts with questions such as “Is this a good option for me?” Shelf Aware establishes who the product is for and how much they intend to eat, then explains the relevant nutrition facts. When the shopper picks up another product and asks, “What about this one?”, it retains the first product for comparison. It also distinguishes personal shopping from family shopping and asks for confirmation before adding an item to a shopping list. Shelf Aware focuses on understandable nutrition information and informed choices.

How We Built It

We are building Shelf Aware around a modular pipeline that connects visual evidence, conversation, and deterministic calculations. Perceptron handles product localization and nutrition-label extraction, while Gemini interprets questions, manages conversational context, and explains validated results. Our calculation layer scales nutrition values to the shopper’s confirmed portion and compares products on a consistent basis. The planned mobile experience uses Swift and SwiftUI with Meta’s Wearables Device Access Toolkit to connect an iPhone to the glasses. A Python and FastAPI backend coordinates product evidence, session state, and model requests. A React and TypeScript companion dashboard shows the source label, portion calculation, comparison, and confirmed shopping-list actions. This dashboard also supports transparent overlays for recorded shopping footage and the wearer’s interface is simple spoken audio.

Challenges

The primary challenge was establishing trustworthy evidence from a moving, real-world camera. Small nutrition labels, curved packaging, glare, and nearly identical product variants made recognition difficult. Shelf Aware therefore needed to identify uncertainty, request a clearer view, and maintain short persistent memory. Wearable interaction adds another challenge: camera capture, microphone input, and spoken output must work together without losing conversational context. We are also designing for interruptions, connection loss, speech-recognition errors, and recipient changes so that an answer about one product or household member does not carry into the wrong decision.

Accomplishments

Our strongest accomplishment so far is turning a personal problem into a product experience. We have developed an end-to-end architecture and interaction plan that connects the exact product, intended portion, selected eater, supporting evidence, and confirmed action. We have also defined clear boundaries between what the models interpret and what the application must verify. Nutrition arithmetic, household separation, and shopping-list confirmation belong to explicit application logic. That gives us a concrete foundation for a useful prototype and an honest demonstration in the next 8 hours.

What We Learned

We learned that a nutrition label becomes more useful when it is connected to the shopper’s actual context. A labeled serving is not necessarily the amount someone intends to eat, and a marketing claim does not explain the full nutritional tradeoff. We also learned that a wearable assistant needs disciplined interaction design. Knowing when to listen, when to ask a question, when to speak, and when to stop matters as much as recognizing the product. Making evidence and uncertainty understandable is part of the experience.

What’s Next for Shelf Aware

We are completing and validating the full product-to-spoken-response loop with a small set of physically verified groceries. We then want to test Shelf Aware with people who find nutrition labels difficult, improve recognition in noisy stores, and expand coverage across products and packaging formats. Longer term, we want to support preferred languages and help shoppers revisit relevant products they have already considered. The focus will remain the same - make health-conscious grocery shopping more understandable, personal, and practical.