Six years ago, one of our team members was a part of a peaceful protest to defend his university's land against government forces. Despite being a peaceful protest, police officers started to pepper spray protesters, fire rubber bullets and throw rocks, injuring innocent people. There was a need for a device to protect protestors then, and there's a need for it now.
Especially in light of many more protests occurring in the world, most notably currently in Hong Kong, Chile, and Lebanon, we wanted to build something that would leverage the power of the crowd to keep these brave people safe.
We introduce CrowdVision.io, a hardware/software smart-glass hack combining the power of having numbers on the ground and an eye in the sky.
What it does
Our platform assumes the fact that many protestors are onboarded with our device and website. Each protestor wears a set of glasses that are decked out with infrared lights, vibration motors, and a camera. The smart-glass' infrared lights are meant to block out CCTV cameras so that protesters can avoid being persecuted for participating. We used a lightweight SqueezeNet deep learning model to help the smart-glasses recognize something violent or potentially threatening (for example if soldiers with guns show up or there is tear gas in the air). The glasses could recognize the danger and alert a web server which registers the potential threat on a map. Immediately, all other protesters in the network within a specified radius of danger are alerted using vibrations in their glasses so they can act fast to combat the danger. The intensity of glass vibration here will be proportional to the closeness of a protester to danger. Moreover, everyone in the area receives a push-notification so they can immediately take action.
On the web application side, we maintain a map of all "danger incidents" near the user's location as well as a running notification bar that provides information regarding the state of the general area.
How we built it
We scavenged a pair of sponsor sunglasses, and started adding our own gear to it. Our motors and infrared lights are controlled by an Arduino while the camera data is processed on our laptop, which contains our deep learning model. From the laptop, if we sense a dangerous scenario, we write the data to a web server which is then updated. This web server can communicate with all the users and alert protester devices.
Challenges we ran into
This was anything but a smooth Hackathon. Originally, we wanted to mount a camera on a Raspberry. We spent hours trying to install Raspbian on a headless Pi, only to be reward with a barrage of SD card read errors. Additionally, we could not find the appropriate adapters to connect to our monitor and we had to give up on the Pi for this hackathon.
During the deep learning model phase, we realized that to do transfer learning on the SqueezeNet would be very computationally expensive and would require downloading huge amounts of image data in order to preserve all classes. We decided to run SqueezeNet out of the box and create additional metrics for measuring the danger in the region.
We originally wanted to run our app in React, but none of us had to React knowledge. This led to many problems along the way, especially since we were trying to incorporate a Firebase server into our protocol. Finally, during a last-minute pivot, we overhauled the entire stack and decided to use PHP instead.
Accomplishments that we're proud of
This was a massive project in scale involving deep learning, full-stack development, hardware, and design. We're proud of the way we fought through the obstacles and successfully integrated all facets of our vision. Even though some obstacles prevailed, we persevered and ended up with a really cool project.
What's next for CrowdVision.io
There are definitely many more features we can implement. If we had more time, we could train a more custom machine learning model to more accurately determine the situation. We plan to use a small powerful computer like a Raspberry Pi or a Jetson Nano that can fit in well with the design of the smart glasses to eliminate the need for wires. Additionally, we plan to implement a clustering algorithm related to the density of danger points that could use a blockchain-like consenus protocol. For example, if many reports of danger showed up on the map, then with higher probability we could conclude that danger did exist there and decrease the number of false alarms.
Finally, there are so many other potential applications for smart-glasses like these. For example, in a construction setting or a warehouse, perhaps the glasses could be trained to identify risk areas that people may not recognize and mark those areas as unsafe when people approach them.