Video credits: Stock footage: Pexels, Music: Mike Noise: Low Earth Orbit
The black summer fires burnt through 24 million acres of land and caused $1.9B worth of damage not including the loss of life caused. This happens every year at some scale and approx. costs the Australian economy $1.5B. These fires are tracked but due to lack of data available as to when they're happening, increases the risk to assets and life.
But satellite imagery is changing that. With close to real-time data being provided, this imagery is saving lives. But, what if it could help prevent the fire in the first place?
Sentinel-2 Image depicting the black summer fires
What it does
A robot that detects forest fires by looking at spectral signatures of vegetation. Ember Bot is mounted with a hyperspectral camera, a camera that looks at light in multiple wavelengths, not just red, green, and blue, and collates this data to give a spectral curve across a range of wavelengths. Since every element and material reflects light in a different way, we can pick up trends in certain material and data and chug them through a logic engine that gets better as it collects more data to learn off of. It's accompanied by a web app that allows for path planning and returns results via a data panel within the app.
How we built it
The Ember Bot
The bot itself is a rover body that's been modified with an aluminum frame to mount the imager onboard it. The hyperspectral imager itself is tech that's been in RnD for over a year and is to be considered an equivalent of an off-the-shelf camera system and does not constitute a major part of the actual hack other than data collected with it. The brain of the bot is an Nvidia Jetson Nano that controls the motors via a relay.
Raw hyperspectral image of a banana taped to a door
The prediction system, mainly coded up in python, relies on deriving and analyzing the hyperspectral image by parsing through each pixel and constructing a spectral curve. We then classify curves for healthy, dry, and burning vegetation. By collecting multiple data points of this data, we train an ML model to classify this vegetation, which can be correlated to the probability that the vegetation can catch fire.
We've approached this classification task in 2 ways. Traditionally logic engines are utilized to compare the output spectrum against known spectrum indices of different substances such as water. These comparisons helped classify certain spectrums and yield endmember concentrations. Our approach uses both supervised and unsupervised machine learning algorithms. In both methods, we undergo data pre-processing, where the hyperspectral image we retrieve is organized into 35 different wavelength bands and their corresponding intensities. In the unsupervised approach, we yield a very high accuracy using both k-means and expectation-maximization clustering. In the supervised algorithm, even with the limited training dataset we still yield great results. As we know these results can only improve over time, as we collect more and more data over land masses not only in Australia but the entire world.
The web-app is built with react and uses the google maps API that tracks the coordinates of the location put down by the user to send to the bot to plan a fire survey.
The app gets the survey direction, beams it over to the bot via wifi, the bot goes out and conducts the survey and runs the prediction workflow on board, and beams back the results back to the app.
Challenges we ran into
Nvidia runs an ARM64 processor system and most of the libs were built for AMD64 which was an annoying thing to work around. Python also ran pretty slow so we had to come up with a solution to accelerate it using the onboard GPU using Cuda acceleration and turning the entire codebase in Cython to compile it using GCC and not run it through an interpreter.
The processing algorithm itself needed threading across multiple cores due to the amount of data it was chugging through. Classification and data collection also mattered a lot on our physical factors such as lighting, etc.
React does not have a good lib for Google maps so we had to use some third-party port (Mapbox) which was shady and took ages to get working.
Accomplishments that we're proud of
Actually building a prediction algorithm that works (at least in our controlled tests). We've gotten 94% accuracy in the number of tests we've conducted which is a shoo-in for government and other commercial applications. All the way from disaster management, to finance and insurance by monitoring assets and investing and divesting based on fire risk analysis.
What we learned
A spherical earth model is actually a bad representation of it and makes it super hard to make maps with satellite and geographical data. But other than that, we learned the full extent of hyperspectral imagery, its use cases, and how valuable it is to multiple different industries.
What's next for Ember Bot - a robot that can predict forest fires
Putting this thing onto a satellite, and scaling this worldwide. Hyperspectral imagery is an up-and-coming remote sensing method and is highly feasible on satellite platforms. The team already has defense, government, and industry contacts and is affiliated with a satellite data company, which means Ember Bot is ready to be taken to market from day 1!
This data can further be used to simulate fires as well. Having risk indices of parts of geography can be overlayed with real-time weather data to simulate the spread of a hypothetical disaster. This information can then be used by the community to put in place measures to prevent the disaster before it has even taken place.