What is Twitter Gang?
Twitter Gang utilizes the power of Twitter to learn more about and visualize the freely-available data about the approximately 33,000 gangs located in the U.S in an effort to improve the lives of those in gangs, and those in the community.
How We Did It
We used Twitter's REST API (though the python-twitter wrapper) to scrape the Follower ID, Twitter handle, and Twitter name of the Twitter of a self-proclaimed gang member, along with his or her geographical location, and stored this data in a .JSON file. We then parsed this file, and visualized this data using D3.JS, overlayed on a geographical map through the Google Maps API. The visualization uses connected, colored "nodes" to signify connections between Twitter users and their followers. Through multiple, recursive iterations on the each of the followers, we can generate a substantial map of both the general distribution of gangs and their relative connections with others.
Challenges We Faced
One substantial challenge that we faced was the API call/rate limitations of the free Twitter REST API (15 API calls every 15 minutes and a max of 5000 data points returned), which we remedied by choosing a Twitter account with relatively few followers for the time being. Another main challenge that we faced was that the python-twitter API was completely new to all of us, but we definitely learned a lot about it throughout our coding and debugging.
What We Learned
We learned that there is a wealth of data freely available through social media services such as Twitter. We also learned that it is important to know how to make the best use of available data.
Next, we plan to scale up our application so that it will be possible to search through more followers and friends and find closely connected groups within those connections. We also plan to geolocation and natural language processing features to find those who are most likely gang members.
We tested various python wrappers for Twitter to find the one that was best suited to our application. We were able to deal with Twitter’s rate limiting by testing our code on followers who had few followers themselves. We also dealt with many syntax issues.