What it does
Shop Vision is a cloud-based infrastructure that minimizes the likelihood of getting infected by combining sensor algorithms, computer vision, and API tracing to detect the number of shoppers, whether shoppers are masked up, whether shoppers are social distancing, and if the occupancy is manageable and does not exceed the maximum. Shop Vision is an app that combines all this so the customer can survey nearby markets/stores in real-time.
Help shoppers navigate shopping during Covid-19 with loosening restrictions and provide businesses with analytics on their store.
How I built it
We knew how complex how the problem we are trying to solve is so we use Microsoft Azure to create our ecosystem. From that, we designed multiple APIs using Microsoft app services that communicate with each store system and each user's mobile apps. With azure machine learning to train a mask detection, social distancing models and to deploy our model. We also created a stream analysis pipeline that passes in real-time data(image frames) into Microsoft blob storage and predicts who is the store is wearing a mask. We have also extracted two other important variables: score and occupancy both of which have their separate API calls hosted on Azure. We used Azure Maps API to get nearby stores given locations. For the capacity tracker, we used an Arduino Mega that had two ultrasonic sensors capturing triggers on either side of "the entryway." Using an FIR filter to clean the signal we then sent this occupancy data to Azure.
Challenges I ran into
None of us were experienced with Microsoft azure. Researching each problems took more time than implementing. The hardest challenge was deploying a real time machine learning model with input video streams.
Accomplishments that I'm proud of
We implemented a bi-model architecture that both predicts people with a mask on and social distancing seamlessly in Azure. Also, we are able to deploy and stream data via azure in two days.
What I learned
Cloud Infrastructure using Azure and machine learning/ computer vision libraries.
What's next for Shop Vision
More graphs and visual analytics