Grocery Manager

• 

  Accuracy and loss of the model

• 

  Confusion matrix of the model

Inspiration

Our original goal was to create an IoT device which could improve our day to day lives. In addition, food waste is an major issue in North America and many other parts of the world, so we thought about a way that we could reduce it by making home appliances and devices more intelligent. Our project strives to make a positive impact on the enviornment as well as improve our day to day lives simultaneously.

What it does

We developed a system which incorporates computer vision with deep neural networks in order to detect when objects enter or leave a fridge or pantry. The device itself will be embedded into your cabinet or fridge and equiped with a camera. It will be connected to the internet and communicate with an API we developed that will keep track of the inventory in your fridge or pantry. With this, users will be able to input the quantity of an item that they wish to keep in stock. Should the number of a certain item decrease below the threshhold, a warning will be displayed.

How we built it

We trained a classification neural network in Keras to identify and label objects. Currently, the model only knows about fruits as this was the best dataset we were able to find for now. We trained the model on a dataset of over eighty thousand images from Kaggle. We split the data into 75% training data and 25% was left over for testing. On this test data, the model was able to reach 97.5% accuracy. The goal of the project was to run the algorithm on an embedded microcontroller, raspberry pi, or a similar device however due to time constaints we were unable to aquire the necessary hardware. In order to run the model in a limited environment such as on an embedded device, the device would cache video onto an SD-card or other storage medium and only run the model once the fridge or pantry is closed and no more new data is collected.

To determine whether an oject is being added or consumed from the inventory, a small OpenCV algorithm would be used to extrapolate the directionality of the event. For example a naive impementation would simply overlay sequential images, ignore 'similar' pixels, and see where additional pixels are added to the overlaid image. Based on the trend of y-position of the pixels, we can determine the direction of the event.

Once all data is proccessed, a request is made to our API which tells it what item quantities were modified and how they were modified. For example if an apple was removed, the API would receive data similar to apple,-1.

A database of approximate lifetimes for each class of items is stored on the server and with this, the server would be able to notify a user of any food which is near it's expiration date. Thus eliminating a lot of waste.

Challenges we ran into

Our first major challenge was that none of the team members had any practical experience in machine learning. We spent much of our time learning how to implement an image classification model in Keras.

The second challenge we faced was attempting to create an OpenCV algorithm to isolate movement and determine directionality. Unforunately due to time constraints we were unable to complete this part.

The third challenge was creating a backend API. We made the API in the Go programming language and this was something none of us had any experience with before.

Accomplishments that we are proud of

We are very proud of our model that runs with a 97.5% accuracy and can accurately classify fruits of 120 different classes.

What we learned

We learned a lot about machine learning and especially deep neural networks. We also learned a lot about creating servers and APIs. We also learned about creating a front end with GoLang and using Jupyter notebooks for scripting.

What's next for "Grocery Manager"

• Send weekly grocery list to user notifying them of what they should buy.
• Label objects with expiration dates.
• Users may order grocery by the click of a button by connecting an online grocery store account or an Amazon account.
• Barcode reading for identification of unknown items.