Inspiration
Our idea stemmed from the fact of us reflecting on the distance learning model, and realizing that we wanted to make sure that students and people that are all at home can have the best experience on these platforms. We realized that health is a huge factor when it comes to everything we do, but with the online model, many people have now subconsciously worked by slouching over their computer or have been presented a large array of false information and propaganda. That is why we created Pandemic.Helper.
What it does
Our project is a 2-piece application that allows users to maintain good posture and prevent back and health complications as well as a chatbot that is able to prevent toxic words and false propaganda. through the use of deep learning we were able to create an application that is able to protect the user and make sure that whether they are in an online class or in a meeting, they are in the best possible environment possible.
How we built it
To develop Big Brother's anti-toxicity plugin, I decided to use a deep recurrent neural network trained on the JigSaw Toxic Comment Classification Challenge data from Kaggle. The benefit of using AI to handle the anti-toxicity, means that we can allow users to casually swear to express themselves, but stop users from becoming aggressive or offensive towards others. To create and train the model, I used TensorFlow 2.4.1, pandas, numpy, and Jupyter Notebook as the development environment. After training the model on its own hardware, I was able to get a model that takes a string as input, and predicts what is the intent of the string as its output. In other words, it converts the input into a prediction of how toxic the input really is. If this prediction is greater than some arbitrary threshold (that the user can adjust), then we will allow the bot to take some action. This is where the second part comes in: creating the bot. Because I used the Python ecosystem to develop the brains of the bot, I decided to continue that trend and use discord.py to create the bot. After reading the documentation, I created a bot that reads every message sent in a server, so long as the bot didn't send it. The bot would then feed the message into the model, and gain the prediction output. Based on that output, it determines whether to punish the user or not. In this case, the punishment is deleting a message and leaving a warning in its place. The bot also logs useful information into a separate channel, with information like the name of the offender, the message, as well as the model's prediction and threshold value.
To develop Big Brother's fact checking plugin was slightly easier than the anti-toxicity, as I used Google's FactCheck API to do most of the hard work for us. I knew that having Big Brother fact check every message would be too redundant and clog up a perfectly fine channel, so I made this into a bot command. I used the requests library to GET the response from the FactCheck API, using a free Google API Key. Then, I parsed through the response, and gained the relevant points, like publisher information, their result, as well as a link to know more. But often times, I'd see that I get more than just 1 of these informational "packets", so I decided to create a paginated embed, which allows users to traverse through all the results, seeing all the responses as well as their links to learn more.
To develop Posture Checker, we started off by training a deep convolutional neural network. Then we quickly put together a dataset of 400 images of our teammate slouching in various angles and looking around his room to try and unbias the data. He did the same but for perfect posture while sitting. The model's design is to classify the posture of someone into 1 of 2 things: good posture, and bad posture. To build and train the model, we used Tensorflow, numpy, and matplotlib. When he had the model ready, we simply used OpenCV to capture the webcam stream, and analyze each frame. We would convert the frame to the appropriate color scale, resize it, normalize it, then feed it into a deep convolutional neural network. With the output of the model known, we'd accordingly draw the correct text on each frame, as well as accompany it with an alarm if need be. Then, we'd display the frame, and loop until the user is done with their task.
Challenges we ran into
Overall, the idea building process was something we spent a lot of time working on as we kept going back and forth regarding the idea we wanted to present and create. Also when creating the dataset for the posture checker, we had to make sure that various pictures from various locations were captured.
What's next for Pandemic.Helper
For Pandemic.Helper we hope to create a more effective combination of our ideas, as well as combine the project with software that will allow the user to view their overall posture from an external app, and see how long and how often their posture changes over the course of a meeting, and then give the user access to this data with an analysis of the data collected.
Log in or sign up for Devpost to join the conversation.