FFFPU (F***ing Fast Food Processing Unit!) is our way of providing a convenient way for you to keep track of calorie intake and other nutritional information throughout the day. It can be difficult and time-consuming to calculate the calories of every meal you eat. This difficulty could even discourage you from logging this information. FFFPU aims to solve this issue by automating this process!
What it does
FFFPU automates the logging of calories and nutritional information from pictures of food that you eat. When you send pictures of your meals to our Slack bot, FFFPU logs the meal, reports information of detected foods, and stores the information for daily summaries of your meals and dietary suggestions. All you need is a blank index card at the upper left corner of your picture for calibration purposes.
How we built it
FFFPU combines several processes: 1) Image pre-processing to calibrate scale and angle for volume estimation via OpenCV 2) Semantic image segmentation via masked fast regional convolutional neural network 3) Image bounding box proposals classified via Google Cloud Vision API 4) Food classification combined with volume estimation to obtain nutritional information via Nutritionix API 5) UI and back-end via Slack API
Challenges we ran into
- Calibrating an image for mass calculations was difficult in the presence of glare.
- Image segmentation failed to work accurately for uncommon camera angles.
Accomplishments that we're proud of
We're proud of how well everything clicks into place. We were able to combine all the various steps - from using machine vision to label and segment foods in an image and calculate volume of the foods, to using APIs to calculate nutritional information and operate a Slack bot - to do this single, well-packaged task.
What we learned
Machine vision can be difficult! There are a lot of different parts to making it work, and there are so many variables and edge cases to account for. Ultimately, we were able to find a way around such edge cases for each application of machine vision we used.
What's next for FFFPU
Faster and more accurate food processing!!