The large outcry and amount of headlines the 2021 Israel-Palestine crisis drew international attention and was a divisive event. Many bias and conflicting opinions were voiced all over social media and the mainstream media and many governments were often accused of being biased.
The inspiration was to find a way to test what the relationship between the way the public and the media portrayed and discussed the events and to see when the limits of bias and subjectivity were pushed.
What it does
This web-app was written as an article and showcases the weeks of text mining and analysis. Millions of characters of text were pulled and analysed and the application discusses our findings in the sentiment, key phrases, and traits extracted from the data.
How we built it
BeautifulSoup and Selenium were used to scrape news articles and the Twitter API 2.0 was used to obtain tweets about the crisis. Expert.ai sentiment, classification, and traits APIs were used to process the data after which Dash + Plotly were used to create the figures and the web-app.
Challenges we ran into
Most of the challenges were around processing the large amounts of data. Word clouds were one of the first proposed ways of trying to find important phrases or traits but this was much too chaotic. This was replaced by simply ranking and separating results and eventually finding similarities between media articles and tweets which resulted in some interesting patterns.
Due to the free-tier limitations the Twitter analysis algorithms were written to be able to pick up where they last left off.
Accomplishments that we're proud of
Being able to obtain the Twitter academic research track based on the idea of this work and future work on this. The beautiful web-app created and the awareness this will bring around the issues of the crisis is also something I'm proud of.
What we learned
A lot of natural language processing capabilities given by expert.ai and how to effectively leverage data processing endpoints and mine/scrape large amounts of data
What's next for Sentiment & Opinion Mining: 2021 Israel-Palestine Crisis
In the future work section we discuss avenues to take this into a research direction to analyse bias or predict public reaction to certain kinds of live-streamed events.
Another avenue would be to generalise the tool and allow a user to input the hashtags and kinds of headlines they would like to analyse and the tool automatically scraping, mining, analysing based on that kind of input.