It is very easy to eat unhealthy and to be blissfully unaware while doing so. Popular restaurant chains side towards hiding important nutritional information. Products package themselves as healthy but in reality, they make a profit off every nutrient they exclude.

We wanted to create an app that makes it easy to eat healthily. One that using the pictures of foods you have eaten throughout the day is able to provide actionable feedback for staying healthy and feel good.

What it does

The prototype of heart tackles deciphering images for the food items. With the app, your able to take a picture of a food item then heart returns the recipe, ingredients, and nutritional count of the food. Then the prototype simply uses that data to count calories for you.

Technical challenges

The difficulty with this app is that computer vision alone does not have enough information within the photo to decipher the exact recipe and ingredients of the dish. We are able to tag the visible ingredients but as there are a large number of variations of different dishes its near impossible to determine which variation is in the picture.

Our solution to reduce the scope of the computer vision challenge was to add geolocation into the mix. We used geolocation to limit the possible dishes that you may have eaten to only dishes cooked by restaurants in the area. This allows us to identify with high accuracy any dishes eaten on going out.

The solution works particularly well for restaurants but not for home cooked food. Home cooked foods are generally simpler; therefore, we maintain a separate database of simple recipes to match against home cooked food that is also partitioned by location to reduce the scope of the computer vision problem.

The app relies heavily on an indexed database; therefore, if a dish is not in the database. We allow the user to add it to our database easily. We load a screen with as many inputs prefilled based on what we think is the closest matching dish.

What's next for heart

The app worked surprisingly well. Our next steps are to scrape data for restaurants in Seattle. Then we will move to production with an MVP on the app store.

Built With

Share this project: