As students, we sometimes live with many other people in one house with lots of Amazon or other package deliveries daily. Some of us have experienced missing deliveries and want a solution to prevent these. We came across the product "Ring Doorbell" however with 9 people living in one house and friends coming over constantly, ring sends notifications for all motions near entrance. We want to identify the isolated events of package deliveries and package thefts. We call our solution Package Cop
What it does
Package Cop is Machine Learning based and detects 3 states of the package: 1) Delivered 2) Stolen 3) Brought Indoors
Changes to these states notifies the user using a text message with the state and a photo.
How we built it
The best way to identify package theft is human behavior tracking. To simulate the scenario of monitoring theft at your door, we built an IOS app to change our phone into a Live Camera Feed. However our ML pipeline can use any video source. Our behavior monitoring algorithm is hosted on a flask server in a GCP Virtual Machine which enhances and reduces the noise of outputs from the Google Computer Vision API. Our Pipeline then responds to the user using a combination of the Twilio API and our Security Dashboard Client which we built with React.
Challenges we ran into
Behavior understanding is not easy. We want it to be real time and accurate as much as possible. We tried Amazon AWS Rekognition at first which has an incompatible feature between its demo and real API. That's when we cannot get accurate object tracking result from image APIs. Then we changed to using real-time images and Google Vision API.
We also need to address the need of behavior understanding, we did a lot of experiment and designed an intelligent algorithm to tell from package delivery, theft and taking away. Server configuration and website designing are not easy. We tried a lot and finally turned to RESTful APIs to be compatible with each other's work.
Accomplishments that we're proud of
We built the first package detection algorithm that detects isolated events of delivery and package removal (delivery acceptance and theft states). Currently the market of cameras and softwares only has motion detection. We are super happy to have done this in a short time and made good friends. None of us worked together before and our team chemistry was awesome!
What we learned
The most important thing we learnt is teamwork. We came from everywhere and had different skills, we made a lot of effort in integrating our work as a full-stack accurate package theft monitoring application. Apart from that, we learnt to play with state-of-the-art machine learning algorithms and computer vision pre-trained APIs for object tracking and human behavior monitoring.
What's next for Package Cop
Scale out our pipeline as a an API for doorbell camera manufacturers and home camera systems.