MoboShop

Inspiration

It all started a couple days ago when my brother told me he'd need over an hour to pick up a few items from a grocery store because of the weekend checkout line. This led to us reaching out to other friends of ours and asking them about the biggest pitfalls of existing shopping systems. We got a whole variety of answers, but the overwhelming response was the time it takes to shop and more particularly checkout. This inspired us to ideate and come up with an innovative solution.

What it does

Our app uses computer vision to add items to a customer's bill as they place items in the cart. Similarly, removing an item from the cart automatically subtracts them from the bill. After a customer has completed shopping, they can checkout on the app with the tap of a button, and walk out the store. It's that simple!

How we built it

We used react with ionic for the frontend, and node.js for the backend. Our main priority was the completion of the computer vision model that detects items being added and removed from the cart. The model we used is a custom YOLO-v3Tiny model implemented in Tensorflow. We chose Tensorflow so that we could run the model using TensorflowJS on mobile.

Challenges we ran into

The development phase had it's fair share of challenges. Some of these were:

• Deep learning models can never have too much data! Scraping enough images to get accurate predictions was a challenge.
• Adding our custom classes to the pre-trained YOLO-v3Tiny model.
• Coming up with solutions to security concerns.
• Last but not least, simulating shopping while quarantining at home.

Accomplishments that we're proud of

We're extremely proud of completing a model that can detect objects in real time, as well as our rapid pace of frontend and backend development.

What we learned

We learned and got hands on experience of Transfer Learning. This was always a concept that we knew in theory but had never implemented before. We also learned how to host tensorflow deep learning models on cloud, as well as make requests to them. Using google maps API with ionic react was a fun learning experience too!

What's next for MoboShop

• Integrate with customer shopping lists.
• Display ingredients for recipes added by customer.
• Integration with existing security systems.
• Provide analytics and shopping trends to retailers, including insights based on previous orders, customer shopping trends among other statistics.

Built With

• google-maps
• google-speech-to-text-api
• ibm-cloud
• ionic
• node.js
• postgresql
• python
• react
• tensorflow