Many people believe that exercise is the only thing that matters, and thus many perform dirty bulking, where they eat terribly but exercise often. This is simply not true, as our diet comprises most of our health, and it is essential for us to watch out for our diet and make sure that we get enough calories and nutrients for our bodies.

However, it is a very onerous task to track one’s calories throughout the day, with many apps out there like MyFitnessPal requiring you to log so much information about your meals before you can eat anything. Wouldn’t it be so much easier if we could just take a photo of your meals and get on with our days? This is what WeightWatcher accomplishes, using Deep Learning and Computer Vision to tracks one’s calories and learn about their dietary habits

What it does

Weight Watcher is a web application that uses Deep Learning to classify the food type present in an image and estimate the weight of the food inside of the image, allowing the model to calculate the calories present inside and automatically log them. Users only need to upload images of their meals and set their calorie and exercise goals to keep track of their dietary and exercise habits!

Steps to use the application

1) Register for an account

2) Create Calorie and Exercise Goals

3) Snap Pics, then eat it!

4) Receive a detailed analysis of your eating and exercise habits

How we built it

It leverages an EfficientNet to perform this task with 95% Accuracy and estimate the weight within 10g error. Models were trained on the ECUSTFD dataset of 3000 images, and trained on Google Firebase, using the Google Cloud AI Notebook and GPU runtimes. Everything was trained using PyTorch and PyTorch Lightning.

The model was deployed on StreamLit and hosted on Heroku here.

Usernames and Registration are handled by MongoDB and the Pymongo library.

Challenges we ran into, Object Scale from images

A big challenge was the idea of Object Scale. It is impossible to estimate the size or weight of an object from a single image, as objects that are 2x larger but 2x far away seem the exact same. In order to correct this problem(How images with an object 2x away but 2x larger look the same from an image), a 1 Yuan Coin is added in the background of the image to give the model a sense of scale. However, we needed to enforce this idea to the model and force it to learn this idea, and thus I pretrained the model for 45 minutes on segmenting out the coin and food, to force it to learn the correlation between coin and food, and then froze those encoder layers. Then, a cross-stitched Linear Head was added on top of the encoder to classify and estimate the weight.

What's next for Weight Watcher

I plan to add more features to Weight Watcher, including calorie burning estimation from videos, allowing everything to be streamlined and no need for logging at all. Right now, users still need to enter how much they exercise, but I plan to add this functionality next.

Share this project: