We were inspired by our university experience from the nights we were hungry but were tired of eating fast food. We just wanted some home-cooked food but we only had random ingredients lying around our kitchen and we’re not really great chefs. As such, we decided to develop this app in which the user takes a picture of ingredients they have lying around and the app returns to them a list of recipes that they can make using their foods.
What it does
This is a native mobile camera app that allows the user to take a photo of ingredients and returns to the user a list of recipes that can be made using the foods detected in the user's photo. The recipes contain names, images, a description, ingredients, directions and nutritional information.
How I built it
The front-end of the mobile app was created on React Native and developed on Expo, so it can be deployed on both IOS and Android devices. It uses expo-camera to take a picture of ingredients and sends this picture using our Google Cloud server to our TensorFlow model created using the Tensorflow Object Detection API. We trained close to 12 ingredients using over 500 images in our model. It then returns a list of foods detected in the given picture and we use our custom python scripts to parse a Kaggle dataset and extract recipes that contain the foods detected. We then take the recipes and their information and put them in a JSON file which is sent back to the front-end app using Flask serving a REST API which is hosted on Google Cloud. The app then lists the relevant recipes for the user to scroll through and pick their choice.
Challenges I ran into
It was a really big challenge to train the model with enough pictures so that it could recognize multiple images in the same photo. Labelling each picture is very tedious and time-consuming. It also took us a long time to get the Google Cloud server up and running as we have never worked with it before. Many issues arose with the app.yaml file and understanding what to put in it and how to use it. There were also a lot of dependency issues with React Native and we had to configure the versions we were using successfully. We also had trouble when it came to navigating from the camera page to the loading screen. In theory, it should have been easy, but until we implemented the Stack Navigator, we failed miserably.
Accomplishments that I'm proud of
We are very proud that we were able to successfully develop and deploy a full-stack application that combines both the server-side and the client-side to create a great overall user experience. We are so happy and proud of ourselves for being able to complete a project of this magnitude within 36 hours. Specifically, we were excited when we were finally able to connect the front-end and the back-end using our Google Cloud server. Getting the TensorFlow model to actually work was also a great success for us.
What I learned
We learned many new technologies in both front-end and back-end development. We learned to use React Native, specifically the Expo Client, JSX, React Navigation and how to navigate JSON data in React Native. For example, it took a few hours just to overload the back button in the header, a seemingly simple task. We also learned to web-scrape using Beautiful soup, how to parse a Kaggle Dataset, and how to deploy our data onto a Google Cloud server to be used in the frontend. We also learned how to use Flask to serve a REST API. Furthermore, we learned to use the Tensorflow Object Detection API and how to train TensorFlow models.
What's next for QuickRecipez
We hope to continue working on this app and one day deploy it on the AppStore and PlayStore. We considered possibly using one of the widely available Food Recognition APIs such as ClarifAI, LogMeal, and Bite. However, these models are very limited and hidden to us, which would provide very little flexibility. Hence, our first step would be to cultivate an image classification model that can detect up to 100 foods to be able to serve the public. We would also need to significantly improve the UI and UX and consider switching to a cheaper cloud platform.