Inspiration We tried to come up with a way that allows people to find out about disaster events worldwide in order for them to intervene if they wish.
What it does Our project is a web app which uses live twitter data in order to monitor catastrophes and classify their urgency based on Natural Language Processing (sentiment analysis). Users can filter the map based on multiple types of natural disasters and time.
How I built it The Front End consists mainly of a D3 application and redux for state management. It is stored statically and served in an Azure Blob and made public.
The backend consists of a VM instance running the main python script which populates and updates the Cosmos DB database, which is run on the same VM and exposed to an endpoint via the Mongo DB interface.
The gathered data is processed with three Microsoft Cognitive services: Sentiment Analysis, Content Moderator and Language Detection.
Everything is dockerised.
Challenges I ran into We also tried using live Facebook data but their API has no streaming and it is essentially designed against scraping. D3 was hard to work with because of its ill-defined DOM manipulation operations. Moreover, state management was also difficult. These were mainly because our lack of experience with these technologies. Exposing the database API from a Ubuntu Server Virtual Host, setting up a virtual host for the Node JS server proving to be difficult. In Python, some of the challenges we faced were character encoding for foreign language analysis and retrieving past data from the Twitter API. Automating deployment to the storage took hours to solve as well.
Accomplishments that I'm proud of The D3 world map. Fetching twitter data live. Making the UI really polished. Using a lot of devOps capabilities from Microsoft Azure. Correctly identifying all relevant tweets about natural disasters.
What I learned Integrating Azure Cognitive services, linking public domain to IP and domain records. Creating public endpoints for APIs in the cloud. D3, working SVGs.
What's next for Team 11 Implementing Machine Learning in order to filter out false positives. Making the app available operate with more languages.
Challenges: J.P. Morgan, Microsoft Azure, Domain.com (howcouldihelp.com), BlackRock