Using social media to categorize information

My code is here: https://github.com/mvz-woyce/bridgehacks-2020

I got this inspiration from just brainstorming about the prompt and thinking about the current problems the world is facing and what solution I can come up with to solve them. I immediately thought of COVID19 and how many people are not aware of the extent to which COVID has spread. My project makes use of the twitter api which allows users to get tweets that have a specific keyword(s) in them. Next, I use a trained machine learning model to classify the tweets into one of eight categories: other_useful_information, treatment, disease_transmission, disease_signs_or_symptoms, not_related_or_irrelevant, prevention, deaths_reports, affected_people. Then they are sent to an auto-refreshing database where they are stored. For the machine learning model, I tried using both and SVM model and a deep learning model and compared the f1 scores for the above categories. The SVM model turned out to have a lower score than the deep learning model, but I still went with it as deep learning models require large amounts of data to give accurate results. The amount of data I had was not enough and so the better accuracy score meant nothing. I used python as the language to build the model and to also stream tweets using the Twitter API. Some libraries I used were numpy, tweepy, and pandas. Some challenges that I faced were when I was building the deep learning model. I had to manually process the data so it could be used. This caused some corruption in the data and it took a fair amount of time to figure it out. The other challenges were simply errors due to different versions of libraries being download.

Accomplishments that I'm proud of

This was my first time in a hackathon and I’m proud that I could pull this off by myself. Using python was also a new thing for me as I specialize in java.

What I learned

I learned to use machine learning and the differences between some types of models and their capabilities.

What's next for Using social media to provide real-time updates

There are many use cases for this project. For example, law enforcement can use this to keep an eye out for any public disturbances, robberies, looting posts, and respond quickly. This idea doesn’t always have to work with something COVID19 related. Any keyword can be used with the Twitter API and with enough data you can easily use this model to create a custom tweet sorting project.

Built With

• numpy
• pandas
• python
• tweepy