While following along the Ukraine Crisis, we wanted to be thoughtful around an underserved population during times of crisis: refugees and innocent civilians. We wanted to build something that could support humanitarian aid organizations who provide help to these underserved populations. Twitter is a great tool for everyday information and situational awareness outside of (potentially faster than) the typical news cycle. And time and time again, Twitter has been a useful communication tool in various crises around the world. Therefore, we wanted to leverage Twitter’s search API to collect this on-the-ground data in order to inform organizations about what’s going on in disaster areas such that these organizations can best leverage their own resources and deliver aid safely.
What it does / How we built it
The project is a Python script that takes in 1 or more keywords plus an optional language parameter. The script uses the Twitter search endpoint to fetch tweets that match these keywords. It also takes retweets and expands them by calling the Twitter tweet endpoint and pulling the full text. When a language is given, the script will translate keywords into the given language using DeepL translator, then fetches tweets. It also then translates the tweets’ contents back to English. The fetched tweets are then sent to Decrypt, Vannevar Lab’s data workflow tool, for users to view the data.
The repository for this project can be found at https://github.com/vannevar-labs/chirp-2022/tree/main/chirp. Access has been granted to firstname.lastname@example.org and email@example.com. The Decrypt app, which is used to display the tweets, and its login information is included in the Github README.
Note: If for any reason the video we submitted does not play as an embedded video, it is publicly viewable at https://vimeo.com/741224051/700d66fced
Challenges we ran into
We wanted to leverage the map functionality of Decrypt to display tweets with geolocation coordinates, but we couldn’t use has:geo in search because the API response is that has:geo is a premium feature, and we only have the basic tier of access. That being said, we’re really excited for V2 to have this feature and to have the chance to try out enterprise access.
fetching retweet full text
Retweets from the Search endpoint are truncated and V2 API doesn’t support “extended_tweet” as a tweet.fields option anymore. Therefore, we also set up integrations with the Tweet endpoint to expand referenced_tweets.id and grab the full text. This wasn’t necessary (we could’ve expanded referenced_tweets.id from the search query), but it gave us a reason to try out more endpoints and dive deeper into the features of the Twitter V2 API.
Accomplishments that we're proud of / What we learned
On the humanitarian front we gained valuable understanding of some of the ways that Twitter provides ways of helping those in need, given a voice to the voiceless in times of crisis. By having Twitter has an easily accessible communication tool, it allows refugees of crisis to ask for help and for charitable organizations to gain situational awareness of what is happening in a chaotic environment.
On a purely a technical perspective, we learned about interacting with the various Twitter API endpoints. We learned that Twitter API v2 has very valuable data enrichments available that improves the ability to filter on Tweets that are particularly applicable to an application. For the submitted app, we used the author_id expansion, but can see how powerful the Tweet annotations could be for filtering as well.
We also learned that with the Expansions, we’re able to get useful related to Tweets without having to make /N/ number of REST calls to do so.
Additionally, we learned that the Twitter API Sandbox is extremely useful for getting started and general data discovery!
What's next for Twitter for humanitarian awareness
We’d love to get Twitter integrated more fully with Decrypt’s additional suite of machine learning solutions, like identifying similar narratives across a large dataset based on more than just string matches. We’d also love to get enterprise access to tinker with those premium features that we couldn’t use for the developer challenge.