SUTOJU

Scan groceries and food with Chefbot's computer vision based interface
Manage refrigerator and pantry content with our web app
Get recipe suggestions for existing items focusing on food that is going old
Measure food waste amount wirelessly with The smart scale
Review your waste accumulation from any time range with the web app
Our architecture combines deep learning, IoT and both local (Pepper) and web APIs (food2fork)

Inspiration

We did some food waste research - it's a huge problem globally.

Global goals
EU platform on food waste
EU food waste statistics
US road map for food reduction
Consumer attitudes towards recycling

We also realized that we don't actually do too much to prevent food waste in our everyday lives. It's a problem that is hard to realize, since it is connected to your daily habits. We wanted to find a way to help people measure and prevent food waste.

We are really passionate about technology, especially machine learning and IoT. We wanted to do something that sounded like an impossible task to finish during a weekend. Our aim was to help people reduce the amount of global food waste to 50% by 2030. We came up with some fun ideas and started coding.

What it does

We created two working prototypes to make reducing food waste easier for consumer households.

The smart scale

Our first prototype is a smart scale for following the amount of food waste produced from a household. You place the scale under your food waste trash bin (or combustables trash bin) and it tracks and displays your habits of producing waste. It helps you realize how much food waste you are producing, which motivates you to follow and reduce your food waste production. It's the fitbit of food waste.

The Chefbot

Our second prototype is a Pepper-integrated application for improving food waste habits in households. It detects the items that you buy from a store and keeps track of the food items at your home. It follows the expiry dates of different items and helps you use the items that are going to expire soon by recommending recipes that you can use them for. No more "Whoops, I forgot something in my fridge and now it's too late."

How we built it

Here's the architecture of the system we built during the weekend.

The smart scale

We bought a typical letter scale and reverse engineered it into an IoT enabled smart scale that is integrated into a trash bin. We gathered its measurements into IBM BlueMix and created an API and a dashboard for viewing the trash bin measurements online.

The Chefbot

We implemented the well-known convolutional neural network AlexNet with Keras+Theano for detecting diffent types of food from images. The deep learning system can take images from Pepper, who can also communicate with the user about the detected items.

We hosted our backend and infrastructure on IBM bluemix, utilizing IBM IoT, IBM Cloudant and Cloud Foundry Apps with Node.js, Node-RED and static website hosting. Our web app is built with Vue.js and Vuex.

Challenges we ran into

Some IBM products were well-documented and easy to use. Some were not. The Cloudant API with the npm package 'cloudant' is still mostly a mystery for us. Building both, an IoT product and our very own deep-learning model for machine vision within a weekend was tough. The Pepper tablet does not support client-side rendered javascript websites well enough, so we had to simplify our dashboard for Pepper.

The image recognition was a huge task. On Saturday evening we nearly lost hope, already building a backup integration to the Watson Visual Recognition API. However, the early hours of Sunday proved successful and we were able to perfect our own solution. In addition to detecting items much faster (because there is no network related overhead), this means our algorithm is portable to offline applications and can be specialized for detecting food.

Accomplishments that we're proud of

Making our product work as a whole. We had many different inputs and outputs between different technologies, but we managed to combine them smoothly into a working product.

We were proud of our Pepper implementation, since we actually hadn't planned on taking part in the Robotics track. However, with the help of awesome mentors, well-documented APIs and enough coffee we ended up with a cool prototype of the Chefbot.

What we learned

We had plans of doing the computer vision as a standalone IoT product with the NVIDIA Jetson we brought with us. We forgot to bring a webcam and the event didn't have any spare monitors, keyboards or mice so we had to replace the Jetson with the Pepper robot. Next time we'll make sure to ask about available hardware in detail in beforehand. Also, we learnt to never give up as the last hours of hacking always bring major advances.

What's next for SUTOJU

We hope to raise awareness and inspire new ideas with our products from the hackathon. Hopefully we'll get to talk to different people about them to see if we could actually turn these ideas into products that make the world a better place with less food waste.

On the tech side we also hope to publish useful open source blueprints for future hackers. Since our project consists of two IoT products as well as the backend infrastructure, each part is a moderate-sized project for tech enthusiasts to start hacking on their own.