Capturing the image screen
The smart fridge
Displaying the recipes screen
Our team after 36 hours of hard work!
Our team has recently been exploring the applications of IoT in every day tasks. Often times, as busy college students, we don't have time to go to the grocery store and keep track of what's in our fridge. It's a pain to go to make something only to realize you can't unless you take another trip to the store. As a result, we came up with a newly improved prototype for a fridge that can tell you what you can make with the ingredients you have.
What it does...
The Smart Fridge of Greatness uses many different new technologies to accomplish its goals. When a user wants to know what to make, they simply take a picture of their fridge. This is then analyzed by a machine learning image recognition program for food and turned into a list of ingredients. These ingredients are then compared with the ingredients for a database of recipes from the RecipePuppy API and turned into a list that is then displayed on the user's Android app in order for easy browsing. The contents of each individual's fridge are still protected by an authenticated username and password system in order to ensure the safety and privacy of our users.
How we built it...
We implemented the Clarifai API that uses machine learning to determine what food items are in the fridge. Since each object comes up as a list of probabilities for more than one potential classification, we decided to just use the one with the highest probability for each object. These classified images are then passed into the RecipePuppy API to determine the list of potential recipes. We permuted through all different combinations of ingredients that used at least 3 ingredients out of the list and displayed all of the resulting recipes from greatest number of ingredients to least number of ingredients. We also used Firebase for the authentication aspect of the app (to maintain security) and to store our data and help the many different technologies communicate with each other. We stored the picture of the fridge, the many recipes, and the login information in Firebase.
Challenges we ran into...
We were aiming for a project that was simple and doable but still ambitious enough for us to learn a lot and step out of our comfort zones. None of us had ever done an Internet of Things hack so it was a completely new experience for all of us. Some of the major challenges we ran into were trying to get all of the different technologies to communicate with each other. For example, we had to figure out how to communicate between our phone camera, Firebase, the python script of recipe analysis, and more. In addition, giving Firebase access to our camera was quite difficult and we ended up having to manually upload pictures instead of having the camera do it directly, as we had originally planned.
Accomplishments that we're proud of...
We're proud of being able to work with Firebase for the first time and learning about all of it's different capabilities that we had never been exposed to originally. We're also really proud of using machine learning for the image recognition aspect of the project and deciding which images could be classified as their respective foods accurately vs others that weren't as strong. We're also really proud of being able to make an IoT hack that could actually be used in real life and that we personally could see ourselves using daily. We also saw potential applications of this technology in soup kitchens and other places where food is often donated and could require inventory and creativity in terms of how to use the resources effectively.
In the future, we hope to continue this project by integrating it with Alexa such that the user could simply ask, "What should I eat for dinner today?" and she could suggest specific recipes that could be made with what's in your fridge that would be suitable for the time of day and/or cuisine one is craving. We also hope to incorporate the camera into the fridge such that the user would not have to take the picture manually but it could be done automatically on a periodic basis (i.e. every time you would eat a meal or start cooking).