Adversarial Learning Is All Semantic Segmentation Needs

Caden Chou, czchou, Hung-Jen Wang, hwang262, Jingyu Tang (Nora), jtang82

Introduction:

The paper aims to use semi-supervised adversarial learning to perform semantic segmentation By combining adversarial loss with standard cross-entropy loss, the paper aims to improve performance of semantic segmentation This model is able to use unlabeled images as opposed to weakly labeled images to enhance model performance Seeing the creative ways in which adversarial networks, which we learned about in lecture recently, could be applied inspired us to choose this paper.

Related Work:

Segmentation network is based on DeepLab-v2 framework with ResNet-10 model pre-trained on the ImageNet dataset and MS-COCO https://www.ibm.com/topics/semantic-segmentation Image segmentation is an end-to-end process that assigns labels to individual pixels to mark boundaries. Image classification determines what’s in the image, object detection determines where in the image the object is located, but we need image segmentation to determine the actual shapes and boundaries. Semantic segmentation, a subset of image segmentation, is a specific type of labeling where the pixels are grouped by their semantic class. This is an important breakthrough because it allows machines to distinguish between object classes and the actual background of an image. Some example use cases include autonomous vehicles, medical diagnostics, farming, and photography. Source Code: https://github.com/hfslyc/AdvSemiSeg

Data:

Forest Aerial Images for Segmentation: https://www.kaggle.com/datasets/quadeer15sh/augmented-forest-segmentation This dataset contains 5108 aerial images of dimensions 256x256. We should not need to do significant preprocessing, since label maps are included in the dataset

Methodology:

Architecture:

The model consists of two parts: segmentation and discriminator networks. The input of the discriminator network is the class probability map, and the output is the spatial probability map. The Segmentation baseline network is adopted from DeepLab-v2 framework with ResNet-101 model pre-trained on the MSCOCO dataset. In considerations of GPU workload, it does not include the multi-scale fusion. Additionally, the last classification layer is removed and the strides of the last two convolutional layers is down from 2 to 1. In this way, the image is resized to 1/8 . As for the enlargement, the dilated convolution in conv4 and conv5 with strides of 2 and 4 is used respectively. Also, the last layer utilizes Atrous Spatial Pyramid Pooling (ASPP) method. Finally, an upsampling layer with softmax is applied. The discriminator network is composed of 5 convolution layers with 4×4 kernel and {64, 128, 256, 512, 1} channels in the stride of 2, each followed by a Leaky-ReLU. An upsampling layer is added to transform the model into a fully convolutional network.

Loss functions: the discriminator network is trying to minimize the spatial cross-entropy loss with respect to two classes. As for the segmentation network, it’s trained to minimize a multi-task loss function with a cross-entropy loss, and an adversarial loss with a chosen weight.

Training:

For the optimization, the model uses Adam and Stochastic Gradient Descent (SGD). The labeled data facilitates the segmentation network with both cross-entropy loss and the adversarial loss of the discriminator network. A confidence map is computed through the discriminator network to train the segmentation network as the supervisory signal with a masked cross-entropy loss.

Challenges:

Data Preprocessing Conversion from PyTorch to Tensorflow A suitable pre-trained Tensorflow checkpoint

Metrics:

Concerning success, our baseline goal is to reimplement the model described in the paper in Tensorflow/Keras. Beyond this, we aim to achieve success in applying our model to a new dataset. The authors of the paper were looking to compare their proposed model against state-of-the-art models such as FCN, Dilation10, and DeepLab-v2. The paper uses mean intersection-over-union mean IoU as the evaluation metric, which measures the overlap between the segmented region and the actual annotated/labeled region from the dataset.

Ethics:

What broader societal issues are relevant to your chosen problem space?

There could be various societal issues relevant to our chosen problem space. Considering how the semantics in semantic segmentation is already on a subjective scale, our model could definitely have biases that may lead to society issues. For instance, different races of people could be semantically segmented into different groups, and this could lead to issues when priority from the labels is required from these labels.

Who are the major “stakeholders” in this problem, and what are the consequences of mistakes made by your algorithm?

Stakeholders could be parties where our work is applicable, and a possible application of this work is towards autonomous vehicles, where the computer system is able to semantically label objects and decide safety decisions correspondingly. A potentially catastrophic consequence is where our model makes a mistake, and a human is labeled into the background. Similar mislabelling can lead to various society issues, and these are all relevant scenarios we should consider for our chosen problem space.