Sophro Shoppers

Make shopping easier through the art of accessibility

Comment

Inspiration

Coming into the hackathon with the idea of a shopping software, we wanted to make a project that is helpful in ways people would never expect. During brainstorming and setup, one of our teammates told us a story that involved one of his middle school friends who was hospitalized after eating stir-fry that unknowingly had peanut oil as one of its ingredients. That’s when we realized that we, as buyers, need to be considerate about the things we buy, because we never know what they contain.

The main problem when it comes to shopping is that not only do buyers spend to the point where they accidentally go over budget, but they also unknowingly buy items that have ingredients that are restricted by their diet. Labels do show the ingredients, but oftentimes buyers would overlook them because of their smaller font size, complex terminology, or the sheer rush of grocery shopping, leaving them unknowingly exposed to health risks.

We decided that for this year's KnightHacks, we would build a user-friendly software that can help make shopping easier through budgeting, macro goals, and dietary restrictions. Want to keep track of your items in the cart? Let our Sophro Shoppers help you practice moderation.

What it does

Sophro Shoppers allows shoppers to get a more in-depth analysis of the items in their cart, while keeping track of needs catered to them. The user will have a tablet and webcam attached to their shopping cart that scans items and informs the user of the macro compositions, the total and subtotal of the items, and whether or not there are dietary restrictions that would restrict items. The user would input their budget and dietary restrictions, change their macro goals, and the software does it for you.

How we built it

React, Vite, Python, and TypeScript are the backbones of our frontend and API. Object scanning runs on Jetson Nano using Computer Vision and YOLOv8 Nano to establish the boundaries for the cart. That way, we can keep track of items coming in and out of it. We also easyOCR to be able to read labels on the items in the cart and classify the brand of the object to give to Gemini for added accuracy.

Finally, the results are transferred into Google Gemini. The API would scour the database and grab the price, macro composition, and dietary restrictions that come with it and return a validated JSON to display in the UI for the user to see. It will also flag the user if an item is found to have an ingrediant that violates a dietary restriction a user has input.

Challenges we ran into

Computer Vision was the biggest hurdle we needed to overcome because it's the glue that held the project together, since it's the one thing that can help us identify the objects placed inside the cart and ensure proper data can be transferred to the Backend and showcased on the Frontend.

We had issues moving from Windows to Linux, especially with how different each of our laptops is in terms of OS.

Finally, starting up the Jetson Nano took way longer than we expected and, as a result, took a majority of our time that could be spent working on other utilities.

Accomplishments that we're proud of

We got the computer vision model to reliably detect the objects we placed in the shopping cart and track their status by adding and removing them, communicating those changes to the Tablet UI screen.

We worked hard on the tablet UI because we wanted users to feel at ease and not overwhelmed. From element coordination to simple color palettes and logos, we aimed to make the backend data as easy to digest as possible.

What we learned

We learned how to use Jetson Nano to empower our project using computer vision, which is another thing we used for the first time. We have also gained experience with using Google Gemini and learned how to prompt it correctly in order to scrape the data we need to properly inform the user about the items they buy.

What's next for Sophro Shoppers

Calorie Counter: We plan to incorporate a calorie counter that would inform the user how many individual and total calories the items in their shopping cart consist of.

Text-to-Speech: Since our project aims to improve accessibility for our users, a text-to-speech option would be essential for those that hae a hard time reading or seeing our UI.

Upgraded Recipe System: While the recipe system is nice to have, we could upgrade it a little more and have Google Gemini scour the internet for recipe links and other items they can buy in order to make those appetizing meals.

Better Mounting Process: Having a Jetson Nano display alongside the mounted tablet and camera would make it so we can use it to roam around in shopping districts and scan more items.

Share this project:

Updates

Submission history