Everyday we see news about how politicians are waffling about, and changing their opinions on topics. How do we quantify that. Is there a possible way to see what extremes politicians have said on a topic, and to call them out for flip flopping using tweets they themselves wrote?
We decided to create an application that would scrape Twitter for tweets by notable politicians, and tweet about issues they have flip flopped. This is an automated solution that runs occasionally in order to check what politicians are saying, and can compare to tweets that have been made earlier. By doing this, we can keep politicians accountable for their opinions in real time.
Notable Technologies Used
For tweets tweeted in the last week, we used Alteryx.
We used Tweepy for updating tweets and getting tweets older than a week old.
We used IBM - Watson in order to actually classify how differing tweets are, and for actually finding which tweets function as good examples of how these politicians change their minds.
We store data as CSVs and pass around JSON objects. All the code is written in Python.
Alteryx provided a good tool to automatically query tweets by specific users using specific words and formats the data in an easily usable CSV, allowing us to build the base on which the tweets 'oppositeness' is decided. Tweepy allowed us to search further in the past and also to update tweets. The tweets are sorted based upon their age, and we can query up to 3200 tweets into the past. This depth of history allows us to get more variation in opinion. Watson, through a combination of sentiment analysis and similarity analysis, has allowed us to figure out how opposite tweets are. We used positivity and negativity vectors that we constructed using such sentiment scores as anger and agreeableness in order to get difference in emotion on a theme (eg, TPP is good vs TPP is bad would be opposite.) This is all done in real time. Tweets are posted back onto twitter using Tweepy.
What we learned
Querying Twitter for large amounts of data takes a large amount of time. Also Twitter provides HUGE objects with a wealth of information already.
What's next for @FullFlipFlop
Have our Twitter bot respond to trending issues that politicians may change opinion on instead of having the themes that it is looking for set by default.