FitTrex

The actual UI tested on a waffle image with the POC app
UI for calorie tracking of food items with the POC app
Concept design for the app

Website!

Inspiration

We're a bunch of fitness enthusiasts and found that most of the available options out there for calorie tracking are clunky, filled with ads and hard to consistently use for each meal. We wanted to develop a nutritional tracking app that is seamless, easy-to-use, takes minimal time to track calories, and personalizable from person-to-person.

What it does

At its core, FitTrex is an AI-powered tool that extracts nutritional information from food using state-of-the-art Deep Learning Neural Networks. Users simply take a picture of their food, draw the box around their food and tap a button, and FitTrex does the rest. It calculates macro as well as micro nutritional information of food, and can identify hundreds of food dishes from around the world. The FitTrex API can also be integrated into existing calorie tracking apps as a pay-per-use service.

How we built it

We trained custom neural networks in PyTorch and deployed on a Flask web app written in Python. The endpoint is hosted on FloydHub, and can be queried with POST requests. The FitTrex app is written for iOS on Swift and allows users to take pictures of their food and estimate the nutritional information of their food. The nutritional information is calculated by (1) estimating the weight and volume of the food item, and (2) looking up a database containing calorific information of food items for fixed volumes, and (3) combining these 2 values together to getting a more accurate estimate of the nutritional information of food. The FitTrex API currently returns the top 10 predictions for food in an image, along with corresponding fat, protein and carbohydrate components normalized to food items weighing 100 grams.

Challenges we ran into

FitTrex consists of a bunch of smaller tasks that were individually straightforward to do; training the neural network, building the iOS app, and designing the UI for the app. The hard part was integrating all of these different parts together to get a single cohesive functional app that works well. The hardest subtasks are:

gathering training data for the neural network
gathering feedback of the app and creating a loop to improve based on feedback
making the Flask web app and understanding the different types of scalability
co-ordinating with the other founders to build the app in our free time outside of school work
learning Swift from scratch

Accomplishments that we're proud of

We're really proud of:

making a working URL that anyone in the world can query to get nutritional information of their food
making a working iOS app that anyone can install on their phone to get nutritional information of their food
training a neural network that can achieve great performance on predicting food classes and nutritional information

What we learned

We learned a ton of stuff building our prototype for FitTrex

Gathering quality training data is hard.
Integrating different technologies is hard
We learned all about the myriad details that go into building a web application and trade-offs that go into designing a functional application
We learned a lot about UI/UX design to provide a seamless and rapid experience to users

What's next for FitTrex

We have a lot planned in store for FitTrex. We plan to do the following:

expand the list of food categories we can predict
improve accuracy of calorie and weight estimation
utilise auxiliary information (such as location information) to get more accurate label predictions of our neural network models
perform more extensive user tests to improve app user experience
develop a beta-ready application built on top of our proof-of-concept iOS application