Note: Please feel free to try our web app out at: https://82cc9ad70f8f.ngrok.io/ Update: This link is now disabled View our youtube demo at: https://www.youtube.com/watch?v=xnaw4taaLoI Our discord usernames are: [redacted] and [redacted]
Recently, in light of the civil unrest, we’ve seen social media become a tool for empowerment. This empowerment, however, should not be limited geographically. We want all people to understand how trends form and continue to develop, and how where you are in the world affects the justice movements you see.
What it does
The user’s experience is relatively simple-- they only have to open an app and then input a trend they’d like to see the search. We take the trend and then analyze it. Perhaps the most important form of analysis we do is sentiment analysis. We trained an LSTM (Long short-term memory) model which is a type of Recursive Neural Network to assign a sentiment score of between 0 and 1. Alongside this, to illustrate where the tweets originate visually, we provide the locations where the tweeting is concentrated: from both parsed out location data and the user’s location if that wasn’t available.
Of course, for parity, we offer a button to view a random tweet in the data set as well as google search links for entities named in the tweets.
How we built it
The majority of our project was coded in Python, with HTML/CSS/JS for displaying the information. We did the Keras/Tensorflow machine learning on Google’s CoLab, only to move it over to a Jupyter notebook. That is, in turn, connected to a Jupyter notebook running a Flask server. We use heatmap.js for the interactive background on the loading page and utilize the Twitter API for the trending hashtags and to access the tweets. Named Entity Recognition comes from the spaCy API.
Challenges we ran into
Because our project has so many components, one of the largest projects was the integration of the features. In particular, once one of our members trained the RNN, we had to figure out how to integrate both Keras and flask. Because of various compatibility issues (including issues with downloading the model and then reuploading it to use), we ended up running the flask server through the Jupyter notebook, which did mean we couldn’t deploy it to a website-- admittedly a stretch goal for us. We also found it quite challenging to decide between various services for what we were doing-- settling on MapBox for the map was a tradeoff we chose to take for its elegant and simple design.
Accomplishments that we're proud of
Because we are both visual learners, we wanted to make sure that we were able to understand the data visually. Therefore, instead of offering numbers, we gave visual representations. The heatmaps, in particular, were a great way to cover large areas of the map. The map also includes the entire world on it, to avoid limiting content and social trends to just the continental United States. This also meant making a fun interactive landing page that the user can draw on with their mouse, and displaying sentiment as a heatmap as opposed to just a collection of raw numbers. Alongside this, we are glad we were able to offer some insight into what kind of information we gather. We are happy that we can give users the ability to randomly access a tweet from the set, and that we gave people the ability to search the terms that came up.
What we learned
We learned a lot about NLP (Natural Language Processing) through this. Initially, we wanted to use ML for the Named Entity Recognition, only to find a better dataset for sentiment analysis. Overall, we learned just how much information can be extracted from a text, and how much data there is in a tweet. From just one sentence, we were able to pull out the locations in the tweet, named entities, and the sentiment available in the tweet. On a more technical level, for machine learning, we learned about how an LSTM can better capture long-range dependencies and be very effective in NLP for sentiment analysis because of the multi-layer structure that lets them store information for longer.
What's next for Movemnt
First and foremost, we want to put this out for everyone to use, and that means deploying it to the web. While there is a tunnel that can be used to access the site, greater access means more people can use the product. We also hope that the app is used for social justice initiatives, and we believe it has promise as a research tool to understand how trends may be confined geographically. When socially aware people use our app, they can see how far their trends and initiatives are moving.