Prithvi4QR

What inspired us

All around the world we observe the impact of climate change, resulting in large scale disasters like the recent floodings in Europe. A main challenge for decision makers is the effective allocation of resources for which they need fast and accurate spatial information of the surroundings. One of the key resources for gathering this information are optical images from satellites, airplanes or drones. Using the labels from the LandCover.ai dataset we saw the opportunity to finetune an Earth Observation foundation model that can support decision makers in quick-response teams by reducing the time needed for manually interpreting large amounts of data.

What it does

The Prithvi4QR model takes optical images of size 512x512px and creates masks including the five classes: background, woodland, water, buildings and roads. It is based on the Prithvi-ViT-100 Foundation Model (NASA and IBM) with a UperNet Decoder.

How we built it

First, we finetuned multiple models on Prithvi and Satlas foundation models and compared them using the JaccardIndex of our three main classes of interest: Water, buildings and roads. Given the time constraints we chose the Prithvi model as our backbone to further finetune on our task. To train the models we used the terratorch, satlas_pretrain, torchgeo and pytorch-lightning libraries. The resulting Prithvi4QR has been trained for 50 epochs on a NVIDIA Geforce RTX 2080 Ti with a changing learning rate, decoder and class weights to improve the strong imbalance of classes in the dataset. We trained 12.1 million parameters (freezing the Prithvi-ViT-100 backbone).

Challenges we ran into

Finetuning the Satlas 'Aerial SwinB SI' was challenging because of very long training time compared to the Prithvi model (8min to the 1min mentioned above). A fair comparison between the two backbones was not possible.
Bringing the model in an operational state was not possible for us. Already, the extraction of predicted masks was challenging for us.
Since the Satlas models to not come with a specific library like Prithvi it took more time to adapt the model and the training.

Accomplishments that we're proud of

We are very proud of us being able to finetune two EO foundation models for the first time achieving notable results.

What we learned

Working with higher level frameworks for PyTorch(lightning, terratorch).
Implementing the Prithvi and Satlas models.
Finetuning and provision of results under time constraints.

What's next for Prithvi4QR

Running additional experiments to achieve better Jaccard Index for road and buildings
Adding more sophisticated augmentation methods (e.g. edge detection image enhancement)
Reducing the parameters using for example LoRA.
Adding additional training data from drone datasets and VHR satellites to enhance the capability of predicting mask from VHR images (since the Prithvi models are pretrained on images with coarser resolution.)