We've been eyeing IBM's Chef Watson API for a while and have been wanted to use it but never got the chance to. We thought MangoHacks would be the perfect opportunity to give it a shot and see what it could do. Unfortunately, we were unable to obtain authorization tokens so we had to look for an alternate route. We thought we could use IBM's image recognition technology to identify types of food and generate information about them. Disastrously, we were unable to obtain API keys and found out that everyone was having trouble using IBM's API. Instead of waiting around for it to work, we began looking for other APIs and services. While scouring the web, we began putting together bigger ideas for our hack. Eventually, we arrived at the conclusion that we would make a smart refrigerator capable of communicating with you and showing you healthy eating options.

What it does is a machine of many talents. We took a boring old mini-fridge and took it to the future. With a touch screen interface on the outside door, and a camera mounted inside, we were able to pull of food recognition and a recipe generator. In the home screen of the fridge, you're able to see the recipes you can make with the food you currently have inside, monitor the temperature, view your calendar, and even communicate with the fridge via voice commands.

How we built it

We build the interface with HTML, CSS, and Javascript. We had two servers which both ran Node. The server and client communicated via Sockets and populated with jQuery and Ajax functions. The camera and sensors on the fridge were connected to an Arduino which communicated with the Edison and then communicated with our main interface server to show the results.

Challenges we ran into

Finding a good API to use, training a custom image recognition model with 4,000+ images, having the Arduino hardware communicate with the Edison Server and then communicate to the main server which then populated the interface real time, and last but not least, having custom recipes be generated based on items you had on a picture.

Accomplishments that we're proud of

Accomplished hardware to software communication wirelessly across two servers, custom image recognition model, and real-time updates to our interface.

What we learned

So many things...

Share this project: