RECOVER-EASE

Instant disaster recovery response and triage at everyone's finger tips.

Comment

Inspiration

Born and raised in South Florida and having experienced hurricanes there and on the North Carolina coast, the damage and destruction is immense.

When the storm ends, there is no time to waste before beginning the recovery.

According to NOAA, each one of these events is responsible for thousands of deaths as well as an average of 21 billion dollars in damage.

The SBP states that having the ability to quickly understand the extent of damage to individual homes enables a more rapid and equitable start to the recovery process, which can save millions - if not billions - of dollars in economic impact.

When the recovery begins, boots on the ground are deployed with a cell phone in hand. Rather than calling various organizations for unavailable utilities, down trees, flood damage, requiring multiple stakeholders, our submission (RECOVER-EASE) can handle all of the requests based on images of the damage - routing it to the proper place.

Getting a wholistic view of the damage is necessary to prioritize and optimize a comprehensive response.

What it does

Our system empowers anyone to document and triage what they are seeing with no prior experience or expertise. We predict who is responsible for the recovery based on an image. This, enables a plumber to help route the utility company to prioritize a downed line or a tree removal specialist to report a fire.

We convert a manual, time-intensive process into an instant, time-saving automated system.

1) We detect the damage in the image 2) We grab the user’s location 3) We predict the responder based on the image: Fire, Structural, Flood, DownTree, ElectricUtility

Anyone can document and report an issue with a single tap and the system triages the content and routes it to the most appropriate responder.

How we built it

It was crucial for this to be accessible over web and not require additional apps or software to allow it to be used by the largest group of users without barriers. For that reason, a webapp was created which leverage native camera function - for instant field images - as well as allowing for uploads - from a laptop at the end of the day.

We started with the DenseNet-161 model and fine tuned it with training images that we found and labeled. Once we were happy with the prediction results and accuracy, we saved the model and deployed it to EC2 wrapped in a simple API that accepts images.

The EC2 instance pulls the model from an S3 bucket and can work independently of the UI. For example, testing was done via postman and it was work with the recover-ease UI or integrated into an existing front or backend.

Challenges we ran into

Finding and labeling the images was time consuming. We still have some issues with the model related to down tree and broken utility poles. Additional time and further tuning can improve this.

A picture is worth 1000 words and therefore find one use case was challenging. We initially started with rekognition and mapped pre-existing labels to the disaster recovery outputs we were looking for. That reached its' limitations as the number of images increased. Which led us to create 5 custom labels and train our own customer model with around 100 images of each category.

In the confusion matrix, we noticed that there is still room for improvement with cracked trees and down power lines given their similarity. In many cases, there are also trees and power lines down in the same image.

Accomplishments that we're proud of

We’re proud to have gone from a sagemaker studio lab training notebook to an end-to-end application and end point that can be accessed and deliver value to a first responder.

Decentralizing the command center was key for this to work. Requiring a centralized dispatch and buy in from everyone for this to work would require a level of buy-in and coordination that would be difficult to get in the early minutes and hours of a disaster relief. As such, we route the content to any email address. Every public utility and relief organization has a public email which can be pre-loaded. If / when a dedicated contact or notification point is available, that information can be updated. Additionally, multiple responders can be added to the distribution list as they become available.

The scope of disater damage is overwhelming. We’re happy to be able to support 5 different organizations and responses which cover the large scope of damage from flooding to fires. These classification allows us to route information to everything from city / town management to first aid organizations like FEMA.

Having a single app capable of transcending various trades and subject matter domains. We’re able to route issues and images to the right person regardless of who documented it. The power company can triangle and report storm water damage. A water rescue team can report a down power line. The telecom can report drywall issues.

What we learned

Everyone on the team was new to sagemaker studio lab. The publicly available examples were great and learning to build on those examples allowed us to focus on the model and not the AWS Sagemaker learning curve. The examples were easy ti deploy by clicking open in Sagemaker Studio Lab and the resource constraints of 12hrs and storage limits were enough for our purposes.

As part of the end to end solution, the UI allowed us to learn the basics of beanstalk and deploying a node.js webapp. Similarly, once the model was trained in sagemakerstudio lab, saving it and deploying it to a flash API hosted in EC2 was a learning experience for most of the team.

What's next for RECOVER-EASE

Impact!

Productization of the experiment with possible real-world uses in it’s current state: The models can be used by the SBP and first responders to audit and verify submitted requests and to detect possible issues in error-prone manual processes.

Edge deployment:

Depending on the availability of cellular networks and internet connectivity in the disaster area, this technology can be moved to the edge to have the models and classification run on the device rather than relying on a spotty cell network.

Video support and multi-issue detection:

Expanding beyond picture to allow video analysis as well as supposed for multiple categories in a single image. For example, an image could have flooding the in one part of the picture and down trees in another. Both agencies and first responders should be notified.

Ensemble Models

The model can also be expanded to include more categories and other damage. It can be used in conjunction with other models that are trained to determine the extent of the damage: Affected Minor Major Destroyed

Built With

Share this project:

Updates