This weekend, my friends I came together to participate in a hackathon. We'd never participated in one before, and only one out of the four us had even done a hackathon! We had a lot of fun and actually managed to build something!
This is a fictional solution to a real world issue showcasing how machine learning can be applied to solve security issues and reduce the need for manual labour.
We highly recommend checking out our video ;)
As part of our project, we wanted to tell a story. In many Hollywood action movies, you see a villain trying to invade a building. For our project, we put a twist on this and instead of having an army to protect us we'd use machine learning 🤯
What it does
The Nerfanator is a modified nerf gun, that fires fake ammunition at any human that it detects through machine learning. There is a camera system that process video from the area. When a human is detected, the Nerf gun is then told to shoot at the unexpecting intruder. The owner of the property can sit back and relax whilst watching the action unfold through the web dashboard. The web dashboard keeps track of number of intruders caught as well as allowing the owner to look back on previous casualties.
How we built it
There are many different components to this project.
We have the:
- Raspberry Pi Camera Monitoring System
- Web Dashboard
- The Nerf Gun + it's triggering system
The Raspberry Pi Camera Monitoring System is using a No-IR Pi Camera. It is running a Python script to create a web server that other device can connect to. It uses the Motion JPEG format to do so.
The Web Dashboard then receives that data from the camera system and analyses it using a Tensorflow.JS model that we trained using Teachable Machine. It's a Neural Network with two classes:
person-detected. The model is running constantly. It uses a combination of ml5.js and p5.js for the processing in browser. When a person is detected it sends an HTTP request to the NodeMCU's web server (more on that later!). It also transforms the p5.js canvas into a data URL that it saves, this data URL is then shown on the right corner where all the images of previous intruders are displayed. It also adds up the count of intruders.
Lastly, we have the NodeMCU + modified Nerf Gun. Here we have a NodeMCU that starts a web server with an API endpoint labelled as
/fire. When we send an HTTP request to
/fire it then turns a servo motor 72 degrees. The servo motor is attached to a copper wire which is hooked onto the Nerf gun trigger button. When turned it pushes down this button. Then when it returns to the origin it pulls up the trigger button which stops the current flow. You can see this below.
Challenges we ran into
WiFi: the WiFi connectivity was not as strong as we thought it would be, hence we were not able to position the product at our desired location. We tried many other locations, and that took some time away from us. This was a challenge as we had to make sure that the camera system was connected to the same WiFi network as the device, otherwise it would not work. After a lot of time we were able to find thee perfect location for our product.
Precision in hardware: to automate the modified nerf gun, we needed to find the perfect angle for the servo motor to be connected with the trigger for the electricity to flow and shoot fake ammunition. This was a difficult task as we were dealing with delicate hardware that could be destroyed easily. We had to take precautions for the hardware and create our system in a way that would be precise enough to work.
Machine learning: We found that machine learning was quite complicated to use as we wanted an external camera to detect the people walking, this was a difficult task as it was our first time using this software for such type of project. This challenge took some precious time away from us, but at the end we got it work.
Accomplishments that we're proud of
We’re very proud that despite this being our first hackathon for almost all of us and that we’d all never done something with Machine Learning we were able to come up with a system.
Specifically, we’re proud of the system that triggers the Nerf gun. Whilst it may seem simple it required a lot of trial and error as well as problem solving to hit the sweet spot.
We’re also proud of our unique idea and the presentation video that we created!
What we learned
As a group, we learn’t about how neural networks work and how image classification works. As this was our first time doing Machine Learning there was a lot to learn (and still more to go!) and we spent an hour or so at the start simply understanding the key concepts.
Lastly, we learnt about the ESP8266 and the NodeMCU as no one on our team had used this platform before.
What's next for The Nerfanator - The Ultimate Home Protector
We are really proud of our product and we would want to continue to develop it as time passes. In the future, we would like the shooting from the automate nerf gun to be more precise, as that would allow the target to hit and for the property to be safer. Additionally we would like to add movement to the gun, we would like the camera to detect the movement of the target and for the gun to move accordingly, this will make our product better as there will be more damage to the intruder.