QUICK-INV

Inspiration

We were inspired

What it does

Our project uses a camera and Google Cloud's Vision API to first take a baseline image of say a shelf or a refrigerator to compare against the current state of the items in the image(whether they are present or not). In addition, we integrated our project with Twillio so that a user could send text requests such as "What's missing in my fridge?" to get a list of items to buy when they're at the grocery store. Twillio can also send the user what's currently in their fridge/cupboard or provides a help function that shows a list of all the commands to use.

How we built it

How we built it: We built this application using a combination of Google Cloud hosting, Google Vision API, and Twillio. We hosted a LAMP server on the google cloud instances that would hold not only a web user interface, but would act as a file server where the remote camera can upload images for processing. This way we were able to process the images as well as store them for later use if we wanted.

The remote camera runs a python script that takes a picture after a set amount of time, that is configurable by the user, and then the device forms an SFTP connection with the server to transfer the files. The server will then check the folder that contains images and send all available images to be processed by the Google vision API. The results of this contain confidence values of what objects are in the image, this data is then submitted to the database in order to maintain a record over time.

We then implemented the Twillio api so that a user can request more in-depth data about what the camera contains, and is missing (from the base line), this is useful for many purposes and is scalable . As a final touch we have data over time displayed on the PHP website.

Challenges we ran into

While working on our project, we ran into several unexpected challenges that took many hours to recover from. We accidentally posted our api key to our GCP services on github where it was picked up by a web crawler. Shortly after, our project on GCP was hacked and multiple delete-protected VMs were spun up to perfrom crypto mining which led google to suspend our project until our appeal was accepted( after 2 days!). We recovered from this set

Accomplishments that we're proud of

We are proud to have a functioning product that does what we initially wanted to do. Further more we ran into several challenges as mentioned above that we are proud to have put behind us.

What we learned

We learned a great deal about Google Cloud platform and the services that were available to us through there. We also learned how to use the Vision API and store & display the various data we collected using a data visualization tool called high charts.

What's next for QUICK-INV

In the future we can add google home implementation where users can use voice commands to check their various inventories.

Built With

• camera
• google-cloud
• google-vision-api
• highcharts
• mysql
• php
• python