"Today, you might Instagram your food. Soon, that snap might help you get healthier."

We live in a highly health-conscious society. Maybe you make an effort to make home-cooked meals everyday, or count the calories and sugar levels in food products you buy. Or maybe you're a college student like us, standing close to the bottom of the spectrum with meals consisting of a rotation of the Triforce — Pizza, Instant Ramen and Chinese Takeout from UberEats.

Regardless of where you are on the spectrum of healthy eating, looking out for your health by being aware of the nutrients of your food is essential — but it's not easy. Not everyone has the time to check the nutrition facts list on everything we buy, or to consider healthier alternatives when McDonald's is the only place open at 3:AM.

"Ditch the text log, and get the full picture of your diet by snapping a picture."

That's why we created Nutriscan — our application lets your take a picture of your food before eating it, and analyzes its calories and macronutrients, letting you keep track of your health and nutrition intake without the hassle of Googling items by hand. Regardless of what your goals are — to lose weight, increase muscle gains, adopt a balance diet, or simply want a deeper insight into your eating habits, Nutriscan will make your life simpler.

With Nutriscan, we want to be more than an application that simply tracks your diet.

We want to prevent people from developing eating disorders, and help those who are already suffering. At least 30 million people of all ages and genders suffer from an eating disorder in the U.S., which is reported to have the highest mortality rate of any mental illness, with at least one person dying as a direct result every 62 minutes. The mortality rate associated with eating disorders is 12 times higher than the death rate of ALL causes of death for females 15-24 years old. The most widely known eating disorders are anorexia and bulimia, which distort people’s perception of their bodies into thinking they are overweight when in fact they are underweight. More people suffer from eating disorders than what's commonly assumed. The majority of these people don't even realize that they have an eating disorder, because anorexia and bulimia are generally considered an uncommon illness found in teenage girls.

I became anorexic and bulimic in high school as a result of body shaming. It’s often more dangerous for men like myself to develop eating disorders because of the widespread stigma that eating disorders only apply to women, and the generally low level of awareness and education regarding eating disorders themselves. I starved myself for a year, generally eating one or two bananas or a small bowl of cereal each day — not really knowing why I was doing this but gripped by the fear that I would become overweight unless I did. I ended up learning about anorexia through a friend who noticed my symptoms and helped me recover, bringing me food and offering emotional support for months.

Men are at a much higher risk of dying from anorexia because they are often diagnosed later due to the stereotype that men can’t have eating disorders. Clinicians believe that many men are ashamed to admit that they may be suffering from something thought to affect only women. Without treatment, up to 20% of people with serious eating disorders die. With treatment, the mortality rate falls to 2-3%. Despite the desperate need for prevention and early treatment, only 1 in 10 people with eating disorders receive treatment due to the lack of awareness and education about eating disorders, social stigma and high cost of treatment. According to eating disorders statistics, about 80% of the girls/women who have accessed care for their eating disorders do not get the intensity of treatment they need to stay in recovery. The average cost for a month of inpatient treatment is $30,000, and it is estimated that individuals with eating disorders need anywhere from 3 to 6 months of inpatient care.

Nutriscan detects potential symptoms of anorexia and bulimia by analyzing our users’ data. It identifies patterns in the users’ eating habits such as the average number of calories consumed and repetitive non-consumption of foods containing carbs or fats. Once a certain user has been identified as showing symptoms of an eating disorder, the application will ask the user specific questions for a more accurate analysis of the users’ symptoms, and provide resources to guide the user through the recovery process.

Yes — we want to help people reach their fitness goals and develop healthy diets, but more importantly we want to combat unrealistic ideals for body image, provide a safe and comfortable platform for people to become aware of the state of their health, and help those who show signs of eating disorders in order to treat them from an early stage.

What it does

The application allows users to take pictures of what they eat in order to understand what nutritional value there is in their food. The immediate result of this is that the user is able to see the number of calories and the nutritional value of what they are eating. But, it is the long term use of the app which really holds encapsulates the value proposition of the application and its societal impact on an individual level. Users will be able to see and receive the following features:

1) Weekly summaries of their consumption habits

2) SMS Alerts if they haven't met their required calorie threshold, or if they have exceeded their daily threshold

3) SMS Alerts if they are consuming food that are in excess of their diet requirements (High Protein, Keto, Low Fat).

These features enable users to visualize their immediate consumption information, as well as the long-term dietary habits and develop a healthier lifestyle.

How we built it

The front-end was built entirely using React and front-end visualization tools. The backend was built using a Python Flask Application that not only provides a multitude of APIs to the front-end, but also incorporates other backend functionality including integration with Firebase (which was used to host user data) and Google Vision with a custom trained AutoML for computer vision. The calorie and nutritional information was fetched using the USDA API, while the SMS texts were sent using the Twilio API.

Challenges we ran into

1) One of the biggest challenges we faced was working with the Google Vision API. The API only allowed us to process one set of images at a time (a max of 500 images) and this was quite cumbersome to work with, especially since the training time was quite substantial. 2) Figuring out some of the CRUD operations at the backend was quite challenging because of the numerous features that had to be kept track of 3) Interfacing a Javascript based React app with a Python based Flask app was a challenge in some occasions, especially when dealing with sending images over a network. We had to deal with multiple conversions in order to upload the image file and send it to the backend as a byte array. 4) Since a lot of processes to fetch and push large data files (images in particular) from the back end became computationally intensive, the usability and smoothness of the website occasionally suffered. To overcome this, we used carefully placed CSS and ReactJS transitions and animations to mask the slowdown that came with these data handling processes so that the user experience was always smooth and seamless.

Accomplishments that we're proud of

1) We’re really proud of the responsive website that works and looks great on both mobile and desktop platforms. The UI is very smooth and responsive to changes in data from the backend.

2) We’re also very happy with the accuracy and speed of the computer vision food classifier we implemented using Google Cloud. Having the ability to classify the object in an image instantly through cloud computing was very exciting to implement.

3) We’re very proud of how seamless our backend and frontend synced together. After each photo was taken, user data was updated in the backend with the new photo data, which was then reflected in an update to the user account page in our webpage. This rapid synchronization of our front end and back end was very impressive.

What we learned

1) We learned how the different Google Cloud Platform services work, such as Google Vision, AutoML, Firebase, and deployment using GCP. Using these Cloud platforms allowed us to implement computationally intensive tasks like ML with our application without the user needing to suffer from these computations as they were handled in the cloud.

2) We also learned to use the Fetch API to send POST requests to send JSON input to the backend API. Similarly, we learned to parse the JSON output from the API and integrated this output into the frontend with React.

3) We learned how to use ReactJS to effectively create a dynamic website that would update in real time to a backend system that was constantly changing. With ReactJS, we found a library that was effective for syncing the front end to the back end stack of our application.

What's next for NutriScan

NutriScan could potentially become a long term assistant in helping people better understand their eating habits, and how they can improve them. This is especially important for people with eating disorders. We believe that the data and insight collected by this app could serve as a great aid to helping people with these conditions. For example, people who’ve constantly been logging low intake of food, or logging intakes of food at unusual hours could potentially before referred to seek help for their disorder. From another angle, this app could also serve as a sense of personal accountability in achieving one’s health related goals.

Share this project: