MEND (Mask Enforcing Nano Drone)

Inspiration

For this project, we have been inspired by the abundance of people not wearing masks when in a shared study area within the university, such as the Billy B or TLC. We decided that we would attempt to make a program that uses a drone with a camera to tell if people are wearing their masks or not, and if not, attempt to remind them.

What it does

MEND is a drone that flies around a route that you can map out yourself on a GUI we built ourselves, and uses the camera to see whether people's faces have masks on or not, using a neural network we trained ourselves using TensorFlow and OpenCV. It may get close to the person's face, which is why it breaks boundaries, and therefore fits with the design criteria. If it detects that the person has no mask on, when they should be wearing them, it will use the data we have stored in a database with AWS to find their mobile number and email address, and send them a text to remind them. If, when the drone next comes round, the person still has not put their mask on, they will get an email reminder along with an image of them in the act! Finally, if on a third inspection they have not, they get another reminder email, another picture, and who knows who'll end up being cc'd into the email!

How we built it

We each took a different part of the project to work on initially, splitting between the facial recognition/mask recognition, and the drone route plotting. Once the neural network had been trained on all of the training data, it could detect masks with a 99.95% accuracy. We used TensorFlow to actually train the network, and OpenCV for the facial recognition and live computation. For the plotting of the drone route, we used PySimpleGUI to create an intuitive frontend, which could then be sent to the Tello drone, and the drone would follow the route. Once the neural network had been trained, and the drone was able to follow a set path, we moved on to using AWS' Simple Messaging Service, and Simple Email Service, to both text/email the offender. We stored booking information in an AWS DynamoDB database which people's information would go into when they sign up to use the study area. This means that when someone is found to be offending, their phone number and their email address can be automatically sent from the database to the AWS API and used to notify the offender. The final step once we had all the individual parts was the knit the whole program together.

Challenges we ran into

We first tried to train the model on around 600 pictures of people with masks on and off, but the model proved to be a bit unreliable, so we upped the training data to around 15000 of each. The only issue was then it took too much RAM for Harry's laptop, as he has 16GB and it took around 24. This meant all the training data had to be sent to Ned's computer over OneDrive, and then he could run the training. Then, we had an issue with the graph to plot as for some reason it just didn't want to work, which would have been fine if we hadn't spent an hour running the training only for it to not save the model... Needless to say we moved the statement to save the model to before the graph plot! We also had some issues with sending the email, as it took us a while to figure out how to attach a picture, so we can have a picture of the offender as well as a message telling them that they should mask up. This was eventually remedied though. Another issue we faced was connecting the drone wirelessly, while using online web services. Since the Tello only connects over Wifi, we couldn't connect to both the drone and the internet. We managed to get around this by running our programs on 2 different devices. Ned's computer would run the program operating the drone and OpenCV video footage. When a maskless person was recognised, it would update a commands file over direct Ethernet to a Raspberry Pi using SSH. Finally, the Raspberry Pi would handle the requests to Amazon Webservices, and push out the texts and emails.

Accomplishments that we're proud of

Neither of us had ever used TensorFlow or AWS before, so just managing to implement something that worked with these was a great achievement! Also, from being such a small team, we managed to accomplish a lot more than we had anticipated being able to do before starting, so the whole project is something we're proud of! We have only done 1 or 2 hackathons each before DurHack 2020, so we are proud of how well we worked, whilst still having time for some Among Us in the middle!

What we learned

At our last hackathon we did together, we spent far too long being indecisive on a project, it was at the 8 hour mark and we had to change our ideas completely, whereas this time we took maybe half an hour to settle firmly on a project, and then we delegated the workflow well between the two of us. A smaller team of only 2 made it easier to communicate and share snippets of code, or bounce ideas off each other as well, which made the whole process easier. Then, we also had all of the programming! Neither of us had ever really done anything in the project except for Python, so most of it we had to learn from scratch or a very basic level of understanding. We also learnt a lot more about how machine learning works, and how practical it is when it comes to making simple decisions such as mask vs no mask.

Built With

• amazon-web-services
• opencv
• pysimplegui
• python
• tello
• tensorflow