Usually we college students don't know how to cook and want to cook quickly. Moreover, we believe people appreciate to have more variety rather than eating the same dish everyday. So we thought what if we could suggest multiple recipes to users based on the available items.
What it does
This application suggests user all the recipes that they can try out with the items present currently in the refrigerator. The user takes a snap of the refrigerator, the application identifies the items through image recognition and suggests all the dishes that can be made out of those items using recipe puppy food search api, and provide interface to buy missing ingredients if any.
How we built it
We have trained and tweaked Microsoft Cloudvision to identify food items like vegetables, fruits, spices. A user would take a picture of their refrigerator so as to capture the ingredients, which will be fed into CloudVision through our app, to identify the ingredients present in the refrigerator. Once we have identified the ingredients, we hit recipe-puppy API which returns all the dishes that can be prepared using the available ingredients. We also suggest recipes which would require additional ingredients that are not available currently and present the option of buying out them from Amazon. We have integrated Amazon affiliate api into the application which would provide seamless integration of amazon app features into our application. We have developed a mobile app using React Native. so that we could host it on both Android and iOS platforms, while our backend server is built using Python Flask.
Challenges we ran into
Handling communication with server and APIs. Sending images in the message objects. Training the model for identifying the food items as the number of images that we can upload to train the model is limited to 5000, which impacts the accuracy of the model.
Accomplishments that we're proud of
We made a highly efficient application that works on both iOS and Android with crisp UI developed using React Native. A well thought out scalable backend architecture. Integrations with Microsoft cloud vision and Amazon affiliate api endpoints
What we learned
Learned about a very powerful suite of services that Microsoft and Amazon offers.
What's next for Hack the Refrigerator
Extending it to an inventory management system by identifying user history and what are the usage patterns. Identifying the kind of recipe a particular user.