You are what you eat. At least that’s what my parents would tell me growing up to try to get me to make healthy eating decisions. When students go off to college, they are faced with the challenge of feeding themselves. Some students choose to stay on the meal plan to eat from campus, while other, brave students will make the decision to be responsible for buying groceries and meal prepping for themselves. The grocery store is a big, scary place where you have to make a LOT of decisions on the fly. What percent milk should I get? Which bread will last the longest? Smooth or chunky peanut butter? Which granola bars will make the best breakfast item? What snacks should I get? Among many other things that go through your head while grocery shopping. And now, with all the rage of Organic and All-Natural ingredients, how do you know what is really best for you?

Our team decided that we all have a common interest in promoting healthy choices, so this is how we came about our project. As we sat around the table discussing different project ideas, Jeanette started looking at her drink bottle and noticed that there are a ton of weird ingredient names that she had no idea what they were or what they meant. And so the idea came to us: What about a system where you can take a picture of your food label, and it would tell you which of those ingredients pose a health risk?

What it does

Our project goes from a real time photo of the nutrition/ingredient label on your food, then it highlights and lists out which food ingredients are not recommended. We did not get the full implementation of our project running, but we were able to test our Python and shell functions separately.

How we built it

Microsoft Azure’s Computer Vision analyzes a picture of the food label to obtain the text from the Ingredients section. We wrote a script to parse the text into individual ingredients and hook up to a Google API that would give the user more information on each ingredient. We implemented hash table to store certain ingredients and their relevant information from Google (i.e. warnings/recalls, foods with similar but different ingredients) to increase speed for future searches. We created a front end that would present ingredient warnings and alternatives to the user.

Challenges we ran into

Our biggest challenge was connecting the Microsoft API to our Python code. The Slack channel was helpful for use to get assistance form Microsoft employees, but, ultimately, our efforts to connect to the API were unsuccessful. We were able to obtain the ingredients text and parse the text in the shell online.

Accomplishments that we're proud of

Our biggest accomplishment is how well we bonded as a team. All of the team members have done hackathons before, but have never worked together in this specific team arrangement. We really enjoyed ourselves and our time hacking yesterday and today. We are proud of the work have completed thus far, and we look forward to hacking together in the future.

What we learned

We gained exposure in using Microsoft Azure® technologies and how to interface this technology with our specific project. Going forward, we hope to use this and similar technologies to enhance our projects.

What's next for Ingredient Explorer

We did not get the opportunity to implement every feature of this project that we wanted to, but if we were able to build it to completion, Ingredient Explorer would be able to suit anyone’s needs. We would have options for indicating specific allergies, avoidances, or types of diets and cater to each of these needs specifically. Our project would be amazing and continue to be simple to use to pull out at the grocery store and help you decide what you want to purchase by revealing what is actually in your food. It would be amazing if we could implement this to happen during real time for example using Augmented Reality technology, in addition to Computer Vision and Azure Cognitive Services Text Interpretation.

Built With

Share this project: