BroccoliBot

Using machine vision and data to get to know your food

Comment

Inspiration

What it does

Our app allows a user to upload a photo of a meal or snack they ate. Using IBM's Watson image recognition, it determines what the photo is depicting. It then uses a combination of public recipe data and a nutritional dataset from Canadian Open Data to determine the nutritional content of the food.

How we built it

The hack is built mainly in C# and Go. The website back-end was written in Go. The front-end was done using Bootstrap. Our backend database was CockroachDB.. We used C# to preprocess the nutritional dataset and to gather recipe data from the internet.

Challenges we ran into

A major challenge we faced was converting the nutritional dataset from Canadian Open Data into a form we could use. This required .csv parsing in C# followed by some clever database design in CockroachDB. Another difficult problem we faced was extracting recipe data from online. Some site we used provided no API, so manual HTML scraping was required. An interesting problem that both the dataset and the recipe data ran into was parsing ingredient lists into a form we could use.

Accomplishments that we're proud of

We're proud of our CockroachDB database, which managed to contain and correlate large amounts of disparate data in a useful way. We're also proud of our recipe data extraction, which extracts a machine-readable quantity and unit from diverse ingredient descriptions like "1 1/4 cup of flour" or "1/2 14 ounce package of cake mix". Our nutrition data set processing was also a great achievement, as it took raw unprocessed data to useful descriptions of the nutritional content of thousands of foods.

What we learned

When making this project, we learned how to deal with incomplete or conflicting datasets. Another skill we learned was how to maintain several interrelated database tables with foreign keys.

What's next for BroccoliBot

Our next steps for this idea would be a per user tracking of the nutritional content of the foods they ate, and comparison with the recommended daily values for someone in the user's demographic.

Built With

Share this project:

Updates