Overview

Shelf turns your home pantry into a "smart" pantry--helping you keep track of food items, warning you of items to buy, and making shopping trips simple.

Motivation

Home automation provides many avenues for improving people's daily lives by performing many small acts that humans often are not well equipped to handle. However, these improvements necessarily come at a cost; in many cases requiring completely replacing old fixtures with newer — smarter ones. Yet in many cases, a large amount of functionality can be included by retrofitting the old devices while simultaneously integrating with new ones. The goal of Shelf was to design a smart pantry system that could improve upon people's own capabilities — such as keeping track of all the food in one's house — while only requiring a slight modification to existing fixtures.

Design

Shelf is designed to be dead simple to use. By affixing a rail to the back of the pantry door, a small device can periodically scan the pantry to determine what food is there, how much remains, and how long until it is likely to spoil. From there it publishes information both on a screen within the pantry, and via a companion app for convenient lookup. Furthermore, all of this should require only slight modification in mounting of these rails.

Functionality

The primary monitoring device of Shelf is the Android Things mounted on the back of the pantry door. After the pantry's doors are opened and closed, this device uses the two servos--one attached to a pulley and one attached to the camera--to move and rotate a camera that takes pictures of the pantry interior and sends them to the server.

Running with Google Cloud Platform, the server processes these images and then runs them through a couple of Google's Vision API annotators to extract product information through logos, labels, and similarity to web entities. Our service then takes these annotations and again processes them to determine what products exist in the pantry and how the pantry has changed since the last scan--returning alerts to the Android Things device if some product is missing.

Finally, the consumer also has access to a mobile React native application that allows he or she to access their "shopping list" or list of items to refill their pantry with at any time.

Challenges

Initially, we attempted to build the product recognition system from scratch using Google's ML Engine and scraping our own image data. However, running the pipeline of creating the classifier would have taken too long even when we reduced the dataset size and employed transfer learning.

There were also difficulties in using the Android Things dev kits, as the cameras in particular had a tendency to be flaky and unreliable.

Future

A smart pantry system holds much potential for future development. Some areas in which we would like to further add include better performance in product recognition (with the additional time to train our own specific image classifier), enabling users to order products directly from the pantry device, and more insightful data about the pantry contents.

Built With

Share this project:
×

Updates