There are so many news articles and tweets written everyday about so many events and people. People want to find out what's going on in the world around them, and see why it's important and why people care.
What it does
After a user query, we scrap Twitter for recent and popular tweets and display the tweets on a world map to show users where their event is being talked about the most. The map also displays the points with a sentiment color, so you can see groups in the world where people really like/dislike the event. We then use different text extraction and analytics to generate a word cloud to show users the most important topics and entities talked about in the news. We also scrap the newsapi for recent news on the event, and display them and tweets on a timeline, with sentiment attached to each article so users can see how others feel about the event.
How we built it
We used Python and many different libraries and APIs to collect the data and visualize it. For the UI, including all plots, we use Dash to host the server (we are in the process of moving the web app to the cloud and connect it to our domain tweetsin.space). We use the Twitter API to collect tweets, and use Pandas to extract and modify the data. The newsapi was used to collect news articles, and Microsoft Cognitive Services to do entity extraction and key phrase extraction to make the word cloud. The timeline is generated using the previously mentioned libraries as well.
Challenges we ran into
We had a lot of trouble using the Twitter API at first, because no one on the team had used it to this extent before. There is also the issue of rate limiting with the different APIs. Other than that, there were the standard programming/debugging errors.
Accomplishments that we're proud of
We were able to accomplish most of everything we set out to do. The different graphics show very interesting insights about current events, and users can get a lot out of it, so we're excited to have this open-source web app available for users.
What we learned
We learned how to use so many different libraries. This was also a great way to learn how to develop a Python web app, and some serious backend while interacting with the different APIs.
What's next for tweetsin.space
We're planning to add more features in terms of visualizing the data, as well as filtering. One idea we have is to figure out if the query is related to finance, and if it is, we would display a stock graph related to the companies mentioned, and then news articles that appear at the stock's spikes and dips. Likewise if the query is related to politics, we want to implement a model that is able to determine political bias, and display left and right leaning tweets and articles on a spectrum. We also want to finish the entity tagging, which would be shown on the map. Additionally, showing some kind of relationship between articles/tweets would be interesting.