WatchDog.AI

Inspiration

The world seems to be getting ever more treacherous and violent. Day by day it seems like we get more afraid of each other, the world’s future, and our own places in it. But, too often, technology is part of the problem! Autonomous weapons and cybersecurity threats are stronger than they’ve ever been. There are no guarantees.

The team wanted to work against that trend and build a minimally-invasive security system. We found this hackathon to be the perfect time to try. We had to.

What it does

It uses machine learning models we fine-tuned to track violence as it happens (in a three-tiered system). It also logs images of the people involved in the violent actions, and keeps track of who’s who. This allows a security officer to know exactly how and when some violent act occurred, without encroaching on the public’s right to privacy.

How we built it and how it works

WatchDog.AI records through a camera, but doesn’t save video until something violent happens. It then records the five seconds before the “violent action” and ten seconds after. To determine what counts as a “violent action”, we fine-tuned the parameters and weights of a ResNet-50 CNN classification model from Microsoft (~99.7% accuracy). WatchDog then saves the video to our CloudFlare R2 Object Storage database. Then our Flask/OpenCV backend detects how many faces are in the video and saves images of those faces under unique-id labels. These unique-ids are made specific to each person, so that if another violent act occurs, more images at different angles are taken of them and saved under that same unique-id. The surveillance officer will be able to then view profiles of all people involved. In the interest of personal freedom and privacy, we do not do reverse photo lookups to find information online about the people in the video.

Challenges we ran into

We wanted to challenge ourselves to try new technologies such as Cloudflare for AI integration and data storing. The data storage part was especially hard. Another major challenge was integrating all the separate aspects of the project. Yumandy had more experience building beautiful frontend applications and databases in JS, React, and TypeScript. August had more experience in the backend, specifically with Python and neural nets, image classification, and OpenCV. It was difficult, but by the end we figured it all out.

Accomplishments that we're proud of

We are particularly proud of our work fine tuning the ResNet model’s accuracy and in getting the backend to recognize particular people across time.

We’re also proud that we made something that looks good and is effective. It was really cool to go from an idea to a real product in just 24 hours!

What we learned

We learned how to decide what’s essential to a project very quickly. Deadlines are good teachers. We also learned how to determine when exactly to stick through a problem and when to quit. We also found learning to use Cloudflare was really rather rewarding, as was figuring out how to integrate disparate parts of a whole.

What's next for WatchDog.AI

The next addition to the project would be really strong encryption on logins, as well as generative AI that would give a video to text overview of what all occurred in a day. The option to reverse lookup specific individuals on confirmation that they’ve committed a violent act or crime would be an interesting direction.

Built With

• cloudflare
• face-recognition
• flask
• google-cloud
• haar-cascades
• javascript
• opencv
• python
• pytorch
• resnet
• typescript