As a former high school soccer player who was forced out of action due to injury, our team members know that a a major aspect of sports injury recovery is diet and nutrition. Done right, it can allow athletes to maintain as much muscle mass as possible, without adding on too much fat. In today's world of restaurant and fast food though, it can become hard to keep track of calories and the 3 macro nutrients: Carbohydrates, protein and fat, especially when you are on the go.

What it does

Our app uses a coreML model and the iOS Vision toolkit to identify foods in the application, and then return the name of the item along with the macro nutrients for 1 serving of the dish using the iOS ARKit.

How we built it

To identify the foods, we used a trained machine learning model based on this paper: which used CNNs to recognize foods. This model was then converted to a coreML model and added to the iOS application for use. The iOS Vision toolkit was used to feed the images of the food from the camera feed to the application itself. Once the ML model identified the dish, we used the USDA nutritional guide API to obtain the calories and macro breakdown for the foods. This information was then relayed as AR-word bubbles to hover over the foods.

Challenges we ran into

We initially tried using a different ML model developed in Matlab to use for our application, and converting that to the CoreML format so it was usable in our app proved very difficult. It took about 7 hours to convert the model and it still didn't work, so we looked for other resources online until we were happy. We also ran into issues with deployment on one of our machines, and were forced to "switch" between development roles towards the end of development.

Accomplishments that we're proud of

What we learned

We learnt how to convert ML models to the coreML format and parse input to them so they can be used in a meaningful way. This was also our first time using the ARKit framework on iOS.

What's next for EatCam

We hope to add a recommendation engine to the application to suggest foods as alternatives that are either lower in calories or in a certain macro based upon a user's requirements.

Share this project: