One of our team members is allergic to nuts and another is allergic to birch pollen found in fruits. They would always find it difficult and time-consuming to determine what foods were safe to eat without having to manually ask the waiters, request an allergen-free menu, look up the exact ingredients of a restaurant food on the Internet, or disect the information written on the back of grocery store foods. We wanted to make this process easier for ourselves and others affected by food allergies.
What it does
A user can open our Android application and simply point the camera at a food, whether is a McDonalds Big Mac or a box of Oreos. The brand and type of food are determined using an ML model, but the user can also input additional information to specify the food if needed. For instance, if someone points the camera at a McDonalds Big Mac, the ML model will determine that it's a McDonalds Sandwich and from there, a list will be shown of all the sandwiches on the McDonalds menu. The user can select the exact sandwich, and the allergen information will be returned with an analysis of whether the food is safe to eat.
How we built it
First, we used a python script to scrape ~20,000 images of 79 different brands of fast food restaurants and grocery stores from Google Images. The images were trained in the cloud through the Amazon EC2 service using a Tensorflow model to recognize the brand of the food item. To determine the type of food, we used Google's Vision API. To retrieve allergen information, we had to build a web scraper to get details from two separate websites (one for restaurant food information, one for grocery store foods). The Vuforia SDK was used to place AR labels.
Challenges we ran into
This was our first time using Tensorflow and Vuforia for AR, so we had issues with compatibility. Training the Tensorflow model with 79 different labels and 20,000 images in a reasonable amount of time was a huge challenge. To solve this, we trained the model with computing resources in the cloud through AWS.
What's next for Allergy Alert
Polish the UI/UX; Train the model to recognize more brands with more images; Release to Google Play and Advertise; Hopefully, develop an iOS version in the future.