Inspired by the need and pressure in day to day life to stay fit and active, and to lead a healthy and balanced diet, we decided to build a technology which would let us monitor our food and calorie consumption and distribution. With the importance of images specially on the rise in the age of Snapchat stories, photo and status updates and a young populace obsessed with selfies on Instagram too, we decided that the best way to approach this would be via images. Most of our target audience - the youth and working class, who don't have time or energy to waste by going to a dietician or nutritionist to map out a diet plan or a calorie chart for them, can use MapCal easily on the go to make a personalised diet plan! And the medium to be used is friendly too. No need to Google the details of the food you'll be ordering and eating. Most of us are familiar with clicking photos and uploading on social media, hence we designed our bot to take the pictures you'll click of your food and send on the chat with the bot, analyses the image and tell you the amount of calories, carbs, fats, proteins etc you will be having. The phenomenal and lucid WhenHub API makes the user visualize his/her consumption in the best possible manner, so he knows at a glance what all he has consumed in the day!
What does it do?
You open the Telegram app on your mobile or via web. Wherever you are, you just use your mobile camera or your webcam, click a photo and send it. The bot will then analyse the image, and tell the user about the calorie, fat, protein, carbs distribution in the food item. The WhenHub API amazingly compresses this info in a visually appealing form, which the user can take in and understand at a glance. Voila! You can use wherever, whenever on the go! While grabbing a quick bite or sitting down for a snack or in the office, you can MapCal anywhere easily in a jiffy! :D
How we built it
First, we tweaked an existing state-of-the-art image recognition model trained on the Inceptionv3 model to identify food items in images. Then, using the Fatsecret API, we obtained information related to food items, such as calories and carbs in them. This service is then linked with a Telegram chat bot, which allows the user to simply send pictures and log their nutrient consumption through time. Using the amazing Whenhub API, we allow the user to then view all of their consumptions in a beautiful time series form of visualisation. We used mongodb for storing user records and nutrient information in food. The image recognition model runs using Tensorflow in almost realtime on a CPU server, and much faster on a gpu server!
Challenges we ran into!
- What about users who have not downloaded Telegram on their mobiles? And are just using it via Telegram Web? Thus this section of our users will not have any username. So, we added this utility in our bot, hence a non Telegram mobile app user can also use MapCal, on web!
- Though the Whenhub API is pretty amazing, lack of exhaustive documentation makes it a little tedious to analyze different API endpoints.
Accomplishments that we're proud of ( ^_^ )
- Sieving food-based tags from the original tags automatically, which has more than 6000 tags.
- Linking all of the APIs together into one fast and simple module.
- Reusing the Tensorflow graph for repeated predictions, thus saving the overhead of loading the weights file every time.
What we learned
- Using the Whenhub API
- Telegram Bot API
What's next for MapCal?
- We can estimate serving size from the image itself, instead of asking the user.
- Expanding the food database to include brand specific nutrition information.