Medical - Food Safe

Inspiration

If you have food allergies like me, one has to constantly read dense nutritional labels to see if a specific product is safe to consume or not. Even if one does not have allergies, consumers are increasingly engaging with these labels to make informed choices about what they consume. Given the amount of information and how densely it is written, a casual trip to the local grocery might quickly develop into a case of migraine. I built this app to solve this problem.

What it does

My app is very simple. The user first enters his/her specific situation as in what interests them in a nutritional label (e.g. allergens that they are allergic to or if they are diabetic and hence interested in monitoring the total carbohydrates in a product). Next whenever they are considering buying a product, they just take a picture of the nutritional label, and the app magically extracts the information of interest to them and surfaces them as a concise nugget which is neither hard to read or digest.

How I built it

I built the app by first integrating camera access so I could enable the user to take pictures of the nutritional labels. Then I used the Google vision APIs (from the firebase-ml-kit) to extract all the text in those images. The rest was easy as I had to use some regexes to match the the keys of interest to the user so I could extract the corresponding values (e.g. /Total Carbohydrates/ -> 37g). To bring it all together, I needed to understand how to use Android-studio, the general architecture of my app (how many activities and such), do the UI design, and learn how to use elements like check-boxes, toasts, and floods.

Challenges I ran into

I had to make tradeoff choices between local vs cloud based implementation of firebase-ml-kit. The quality from the local seemed just as acceptable as the cloud based solution. So, I chose the local solution as it was free and also saved the user round-trip time to-from the cloud hosted APIs on each request.

To my dismay, I also found that the text that seems to be so nicely laid out on the nutritional labels comes back from the vision API as a pretty incoherent blurb of text. I used the power of smart regexes to identify and extract the specific text of interest to the user.

Accomplishments that I'm proud of

I am very proud of how I used the Vision API in the couple of different ways and made quality, cost, and latency based choices to determine the right way of doing things. Using regexes was pretty innovative and helped me solve the big problem of dealing with garbled text returned by the API.

What I learned

Learned how to use the Google Vision API from the firebase-ml-kit. I also learned a lot about regexes.

What's next for Food Safe

I want to add features like - allowing users to store nutritional labels of what they are buying so they can track consumption based on re-buys. I also plan to build some machine-learning models to more robustly analyze nutrition labels as the inaccuracies in text outputted from vision APIs can be best fixed by predictive ML models. I also plan to offer users the ability to compare similar products based on the information in their nutritional labels.

Built With

• android
• android-studio
• firebase-ml-kit
• google-vision