Inspiration

The inception of Team TerraHackers can be attributed to the common desire to leverage the power of technology to have a positive impact on the world.

Natural Disaster is one of the greatest challenges faced by humanity in Modern history. They impact communities, lives and properties at unfathomable scales, making both the relief and recovery from these hazards extremely difficult. After disasters have occurred, Governments, organizations, and Nations face daunting tasks of rebuilding and restoring infrastructure and assets while ensuring the well-being of affected citizens.

In just 20 years since the beginning of the 21 st Century, there have been almost 3 trillion Dollars of directed losses reported due to natural disasters across the world. Yet, there are many limitations in the Disaster Risk Management sector. A staggering 110 countries (56.4%) do not have approved Disaster Risk Management Strategies. Limited data verification pipelines can cause mass panic among people. For example, the Pacific Early Warning System has a false alarm rate of 75%. Finally, much of the current efforts involve manually going through terabytes of data during sensitive, time-critical scenarios, which often leads to loss of thousands of avoidable deaths. There is almost an absence of the use of cutting-edge advancements of Artificial Intelligence to optimize processes.

The impact of Climate Change on the scale and frequency of Natural Disasters cannot be understated. The number of climate-related disasters has tripled in the last 30 years. Even if the global community fulfills the emissions targets stated in the 2015 Paris Agreement, communities will continue to face the effects of climate change, especially natural catastrophes, for decades to come. This means that, in addition to addressing the causes of global warming, we should endeavor to mitigate its effects and leverage technology to build tools for community resilience.

Hence, came the inspiration for Geolytics.AI – a one-stop solution to solve all the above problems with 3 major features: Disaster Detection, Damage Assessment, Damage Simulations and Organizational Workflows. The main motivation was to leverage the power of technology to build disaster-resilient communities and accomplish the 11th Sustainable Development Goal of Sustainable Cities and Communities as laid down by the United Nations.

What it does

Geolytics.AI will be the next big step forward in accomplishing the 11th Sustainable Development Goal of Sustainable Cities and Communities laid down by United Nations. Our project solves the following UN SDG's

  1. Sustainable Cities and Communities (SDG 11)
  2. Climate Action (SDG 13)

We discussed with National Disaster Management Authority (NDMA) India and Earth Observatory of Singapore to understand the gaps in technology for DRM and collaboratively develop the following features,

Disaster Detection: The first step in disaster relief is detecting disasters. However, there is over reliance on sensor readings for detections, which leads to a single point of failure in critical situations. We developed Cross Validated pipelines integrating satellite and sensor readings with witness reports for accurate detections.

This includes real time Disaster Detection from witness Image and video streams, NLP based Citizen Science for community verification and Hazard Reports from relief portals.

Damage Assessment: Upon detection of disasters, teams are sent to manually assess the damage. This process takes days to weeks and is prone to human error, leading to massive losses in lives and property.

We have developed Deep Learning Systems for Damage Localization and Classification from Satellite Imagery (which is the first source of data in disaster situations) with over 97% F1 score for the State-of-The-Art performance. This completely removes the need for on-site assessments after disasters.

Damage Simulations: We provide hazard-agnostic simulation tools to assess the performance of diverse assets after disasters. Organizations can use this feature to estimate losses to assets, their repair costs and estimated repair times for performance engineering.

Dashboards, analytics and support for getting quick insights and integration into organizational workflows

Reliable and scalable REST API endpoints that expose our AI models and rich backend data for seamless integration

The value addition of Geolytics.AI is tremendous, as it addresses specific pain points of current DRM systems effectively:

a. Artificial Intelligence powered decision making

  • Powerful AI algorithms and backend computations for disaster-specific, real-time intelligent solutions completely eliminate the need for manual supervision
  • Object Classification to record and classify disasters based on witness images in real time, providing accurate, cross validated detections
  • Computer Vision and Image Segmentation for damage localization and assessment provide deep insights about the on-ground situation and copmpletely automate the process of manual post disaster assessments
  • Natural Language Processing for real-time tweet classification and geotagging
  • Simulations and Modelling for various hazards and their aftermath

b. Multiple eyes, single platform

  • An integrated platform with a single database ensures data consistency
  • The same set of information is visible to all people using the software. Gone are the times of miscommunication in time-sensitive scenarios!
  • AI models and validation pipelines run 24/7 to only expose authentic data

c. Organizational workflow integration

For users who prefer to manipulate and use data in their custom way, our highly reliable API services provide simple access to rich data in the industry-standard JSON format. Plug-and-use in this sector has never been easier! All website offerings are available as consumable APIs, namely dashboards, data visualization, live area monitoring and inventory management.

Ideation and Collaboration

Once we had the idea of building a Natural Disaster Risk Management System, we did further research to learn about the existing technologies and reach out to organizations for understanding the current Disaster Management Strategies.

Discussions with the National Disaster Management Authority of India and the Earth Observatory of Singapore allowed identification of Technological Gaps and Time-Critical inefficiencies in Disaster Relief and Risk Management.

Mr. Sebastien Biasse, a research staff from the Earth Observatory of Singapore, told us more about the potential for Software and AI driven methodologies in Analysis and relief of Natural Disasters. He also provided us with resources, journals and publications tackling this topic, which were essential in learning the technical details of scientific analysis required for assessing disasters and impact.

Mr. Mohit Sharma, a Civil Defense leader from the National Disaster Management Authority of India (NMDA), gave us insight into how disaster relief is carried out in India. The NDMA has begun investing heavily in Technology for disaster management. They have set up Geographic Information Systems (GIS) which can monitor and collect satellite imagery and geospatial data of the regions being monitored. Additionally, they are also currently working on developing a mobile application which can be used by the relief and scout regiments to call for help and increase the support radius based on the scale of the disaster.

However, there are no existing technologies to provide intelligence to these organizations. Upon detection of disasters, teams are sent to manually assess the damage, its impact, severity and exact locations. This process takes anywhere between days to weeks and is prone to a high degree of human error, which leads to losses in lives and properties. There is no automation of disaster relief processes, with analysts having to sift through massive amounts of raw data to identify damaged regions. Even the mobile application which is currently being developed by the NDMA requires manual assessments and triggering of disaster support teams.

This lack of AI enabled technologies in Disaster Assessment and Risk Management motivated us to develop our project, Geolytics.AI - A single software to solve all these problems from a scalable platform.

How we built it

The software architecture adopted for Geolytics.AI is a blend of the Client-Server and Microservices architecture. The Microservices Architecture helped avoid a monolithic application and instead suggested 3 microservices as separate modules that could serve the information through a streamlined delivery pipeline, independent of other failures in the application. To serve this information, we decided to use the Client-Server architecture. This facilitated communication between the frontend requesting resources and the server that processed and returned the requested data/content.

On a high-level, Geolytics.AI has 3 parts - Disaster Detection (for detecting disasters and data verification pipeline), Damage Assessment (AI models and simulations to assess damage of disasters) and Organizational Workflows (APIs and data visualization of important statistics). Each part works independently and is solely responsible to provide a distinct service.

The technologies used are:

  1. Backend: Google Cloud, Azure and Heroku (for hosting microservices), TensorFlow (for developing ML models), Flask, Twitter API and ReliefWeb API (for citizen-science), Python
  2. Frontend - Google Cloud (for hosting PWA), React.js, Node.js

All the Machine Learning models were built and trained using Tensorflow and Keras. Hosting the frontend was greatly simplified by using GCP App Engine, which also empowered us with increased observability through metrics such as incoming traffic, request load etc.

Frontend
  1. The entire frontend was made using React.js. We used several concepts like React Hooks, state management using Redux etc. to fulfill the requirements,

  2. The frontend design was inspired from Shneiderman's Eight Golden Rules of User Interface design, to guide us towards making an intuitive and easy-to-use interface.

  3. To enable public access, traffic handling and scalable design, the frontend was deployed on Google Cloud Platform to enable public access and easier judging.

Disaster Detection
  1. Latest Government Information
  • This service used several humanitarian portals like ReliefWeb and USGS Notification services as the primary source of information

  • REST APIs and subscrption mechanisms were used to collate the information from all the sources of data.

  • The backend was developed using Flask, which exposed the "/latestdisasters" API endpoint. This was deployed on Azure, for public usage by the frontend (which just had to make a simple API call to the backend to receive the data and display the results)

  • The backend involved a sophisticated pipeline, that in a step-wise fashion, collated and stored data from various humanitarian portals, filtered out data that did not pass cross-verification checks with the incoming data, followed by sorting and filtering the data in terms of relevance.

  • Web scraping was done to get the latest verified information of each of the disasters collected in the process. Such data was cached for performance purposes, so that the frontend could access it at a later stage.

Disaster Classification
  • The frontend involved receiving the data from the user, encoding it in into base64 encoding and sending the stream of bytes to the backend

  • The backend was deployed on Azure Machine Learning. The base model was the "EfficientNetB7" model, which was finetuned with custom layers and trained on a dataset with 5,000 images of several natural disasters. The test accuracy at the end of training was 93%

  • The model was deployed on Azure ML. The rest of the tasks such as receiving the stream of base64 bytes, decoding it into a JPG / PNG, passing it into the model, performing inference and returning the results were packaged as a pipeline using the Python "Joblib" package and also deployed on Azure ML resources.

  • The frontend was then integrated with the backend that involved the usage of a simple API call to achieve the relvant functionality.

Citizen Science
  • The first part of the backend involved starting a live stream of Tweets by using Twitter API. The tweets were then categorized into whether they talked about natural disasters or were unrelated.

  • The categorizer made use of a custom trained NLP model. The NLP model that was made of a tokenizer, label encoder and a subsequent Keras Sequential Model was trained on a huge dataset of around 60,000 tweets that were collected over the past 48 hours. The test accuracy at the end of training was around 92.3%. The model was deployed on Azure for easy, API-based public access.

  • Once the categorizer filtered out unrelated tweets, it was sent to a geotagging a service, a service built using Flask and deployed on Azure to be used as a public API, that made use of "spacy" and "geotag" packages of Python to get the latitude and longitude of the disaster being discussed in the tweet. Once done, the results were returned back to the frontend for display of information to relevant authorities as well as people near the site of impact.

  • The frontend integrated simply with the backend as it just involved making a REST API call.

Damage Assessment

Damage Assessment uses a combination of a localization polygon detection model and a classification model.

Our initial baseline approach was to try and do both the building localization and damage classification tasks in a single U-Net model. Not too surprisingly, this approach did not perform well due to the model having to express both the coarse-grain task of separating buildings from the background combined with the finer-grain task of rating the building damage. Thus, we decided to split the problem into the two subproblems of localization and damage classification.

  • Damage Localization

We are using a fork of motokimura's SpaceNet Building Detection for our localization in order to automatically pull our polygons to feed into our classifier.

Using a dataset of pre and post disaster image pairs, we trained a U-Net inspired CNN model to segment polygons of interest from each disaster imagery.

Our main algorithm of choice during the competition was the U-Net convolutional neural network (CNN) architecture, which is a popular algorithm for semantic image segmentation.

The U-Net architecture augments the fully connected CNN by having a series of contracting layers that capture context followed by a symmetric group of expanding layers that allow the model to learn accurate segmentation boundaries. The contracting layers are called the down-sampler or encoder, and the expanding layers are called the up-sampler or the decoder.

Our model had a test accuracy of 97%, giving us State-of-the-art performance.

For the localization problem, our goal was to classify each pixel in the pre-disaster image as either "asset" or "no asset"

Once we had our model trained and ready, we used Flask to package the model into a service and deployed it on Azure App Services, with model weights and metadata being stored on Azure Machine Learning. Our code is hosted on github, with GitHub actions as the CI/CD tool for deployment to Azure.

We performed unit tests, benchmarked the performance of the model and set up traffic control and load balancing to ensure reliability of our Service.

  • Damage Classification

We utilized tensorflow and opencv libraries to create tensors of dimension (1024, 1024, 6), representing the concatenated pre-/post-disaster images.

Then, we used the output of the localization model to mask the tensor so that all non-building pixels had a zero value. Next, the tensors were randomly cropped to be of dimension (256, 256, 6), and only crops that contained at least 20% of non-zero pixels were used in training.

The assets were to be classified with a score of 1 to 5, with 1 being no damage and 5 being completely destroyed. This is an example of an ordinal regression problem and we explored several techniques for using neural networks to solve it. Our optimal solution ended up using an ensemble of U-Net models to perform the task.

Our final classification F1 score came to around 81%, yet again giving us SOTA performance.

A similar protocol was followed in the deployment of the Damage Classification Micro-service

Damage Simulation

We provide tools for assessment of damage and losses due to natural hazard events by using a stochastic damage and loss model that is an extension of the high-resolution PEER performance assessment methodology described in FEMA P58 (FEMA, 2012).

We developed a wrapper around the pelican python package to provide modular simulation toolkits for Probabilistic Estimation of Losses, Injuries, and Community resilience Under Natural hazard events

We packaged the developed simulation toolkit in a flask app and optimised it for deployment.

This included caching and file system handling to ensure scalable deployment. Implementing caching was especially important since this micro-service was deployed on Google App Engine, which does not allow usage of file system memory for writing of data files.

We used inbuilt modules to generate sample data for testing for our users and implemented data processing techniques to transfer large csv files to the frontend for display.

Google Cloud Platform App Engine was used for deployment and serving of the service. We have included auto-scaling and traffic handling features to ensure a smooth experience and reliable performance for the users.

Organisational Workflows
  1. The data collected by the disaster detection service was used for rendering of several custom visualizations

  2. Many of the social media and inventory management features are in "beta" but can easily started by plugging in the relevant sources of information

  3. "Matplotlib" package of Python was used for making intutive and effective visualizations.

Challenges we ran into

To build this product, we ran into several challenges, but each of them were promptly addressed to ensure the development of a well-working prototype. Some of the major challenges that we ran into are:

  1. The technical architecture of the entire system had to go through several iterations to ensure that all the systems are coherent. Due to the pipelined nature of the application wherein the information goes through disaster detection followed by damage assessment, we had to figure out solutions to problems like where to fetch information, fastest information source, latency for damage assessment etc. to get properly functioning services

  2. We had to go through the implementation of several ML models for different services (like Disaster Detection, Damage Assessment etc) to ensure that the accuracy of the returned results was high

  3. Since there were so many services, a significant amount of time was spent to come up with the design of the frontend. We had to ensure it was intuitive and easy for the user, and at the same time powerful to complete all the required tasks with the lowest possible latency. We followed Shneidermann's 8 Golden Rules for the User Interface design.

Accomplishments that we're proud of

This project gave us a unique opportunity to experiment, develop and work on an extremely challenging and socially impactful problem. Not only did it give us a chance to interact with professionals such as Researchers, Geologists, Defense Authorities and Scientists to identify and solve problems, but it also helped me explore different technologies and aspects of building scalable applications and complicated pipelines. We are proud of having picked up on so many diverse technologies and bringing them together into a single production ready software which is ready to serve and help people.

What we learned

Building Geolytics.AI was a great learning experience for us, as we researched on a niche area that can make use of innovations in technology. We learnt about the following:

  1. Existing Disaster-Risk Management solutions, their offering and their shortcomings. This helped us find pain points that can be addressed by Geolytics.AI

  2. Existing methodologies by real disaster-response authorities and the technical innovations used by them

  3. The communication gaps between authorities and citizens, and how this gap can be reduced by bringing and spreading verified informtion as quickly as possible

  4. We learnt deployments to multiple cloud platforms like GCP and Azure. We learnt a lot about REST APIs and how every service can be managed easily by containerizing it and exposing it as a public API

  5. We worked with a wide range of tools, technologies and platforms like Azure, GCP, Python, JavaScript, Rect, Node.js, Web Scraping etc. which was very enriching.

What's next for Geolytics.AI

Geolytics.AI is already a Minimum Viable Product (MVP) that can be used by people. Given its great potential, there are several future extensions to this product:

  1. Making a mobile application component for registration by citizens, where they can upload witness reports (to make the disaster detection pipeline more reliable) and receive SOS / Alert notifications etc.

  2. Using AI and ML to perform automated evacuation from satellite imagery by using the damage assessment service. This would make the application more holistic.

  3. Adding scalability to the application by making use of load balancers etc. as it needs to be robust to a naturaul disaster situation where it can expect traffic from thousands of concurrent users.

  4. Upgrading existing services by incorporating more sources of information and using more powerful machine learning models for a more reliable, production-level offering.

  5. We are already conducting experiments for features such as Evacuation Route Generation from satellite imagery, and developing inventory management systems for efficiency in time-critical situations.

  6. We hope to reach out to even more organisations and roll out to the market soon, gaining feedback and improving along the way.

Built With

Share this project:

Updates