VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY FACULTY OF COMPUTER SCIENCE NGUYEN THANH DANH - PHAN NGUYEN THESIS SEMANTIC IMAGE SEGMENTATION IN THE DARK WITH DOMAIN ADAPTATION METHOD HONORS BACHELOR IN COMPUTER SCIENCE HO CHI MINH CITY, 2021 VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY FACULTY OF COMPUTER SCIENCE NGUYEN THANH DANH - 17520324 PHAN NGUYEN - 17520828 THESIS SEMANTIC IMAGE SEGMENTATION IN THE DARK WITH DOMAIN ADAPTATION METHOD HONORS BACHELOR IN COMPUTER SCIENCE THESIS ADVISOR NGUYEN VINH TIEP, Ph. HO CHI MINH CITY, 2021 ASSESSMENT COMMITTEE The Assessment Committee is established under the Decision., date HE 118141111 1e by Rector of the University of Information Technology. Ly Quoc Ngoc — Chairman. Nguyen Thanh Son — Secretary.
Nguyen Hong Phuc — Member. Acknowledgement It is not possible to prepare a dissertation without the assistance and en- couragement of other people. This one is certainly no exeption. Thus, this dissertation is dedicated to those who make it possible: First of all, we would like the express our deep gratitude towards our advisor PhD.
Nguyen Vinh Tiep, who inspired us the passion of research, for his instructions and supports during the process of our work. Besides, we would like to thank the Faculty of Computer Science - University of Information Technology where we have been prepared enough knowledge to pursuing the field of Computer Science research. We also thank Multi Media Laboratory (MMLab-UIT) for supporting us the research environment and cutting-edge devices throughout the university journey. Specially, we want to say thank you to our fellow friends who support us with precious ideas and are always willing to help us when we are in need.
Last but not least, we genuinely thank our families without whose love and support we would not have been here to complete this dissertation at the university. ABSTRACT Semantic image segmentation is a fundamental computer vision task that brings many conveniences to human life. Its applications vary in many aspects such as autonomous vehicles, medical systems, photo editors and so on. For instance in the field of autonomous vehicles, the vehicles take images from cameras along with other sensors to understand the surroundings.
The taken images are automatically handled by computer vision algorithms including semantic image segmentation. In the normal condition of daytime, many previous work were finished with yielding high performance. However, in nighttime condition which we define as in the dark, there are several work done with poor results. To be specific, we discover that the specifications of images in daytime is different from those in nighttime mainly due to light condition.
Furthermore, there is a limitation of annotated nighttime images for semantic image segmentation purpose. Focusing on the application of autonomous vehicles, we do the task of semantic image segmentation on nighttime cityscapes images. In this work, we propose a complete framework to solve the problem of semantic image segmentation in the dark with the help of domain adaptation method. To address the problem of lacking suitable annotated dataset, we leverage Generative Adversarial Network - GAN to transfer images from daytime to nighttime domain.
In segmentation task, we apply self-training method and propose a loss function to enhance the performance of the model. Finally, we show the improvement and efficiency of our hypothesis in a series of experiments and lessons for future researches. ee xa 5 12 Objectives.4 Dissertation Structure 7 2 Related Work 2.2 Fundamental Knowledge in Convolutional Neural Network .3 Semantic Image Segmentation.1 Overview of Semantic Image Segmentation 10 2. Fully Convolutional Networks .5 Pyramid Scene Parsing NeUwork.7 Google DeepLab Family .4 Generative Adversarial Network.1 Overview of Generative Adversarial Network.2 Conditional Generative Adversarial Network .3 Pix2Pix Ha ee 33 244 CycleGAN.5 Image Domain Adaptation.1 Overview Proposed Framework .2 GAN-based Image Translation Component.1 Variational Autoencoders- GAN .4 Perceptual loss maintains the semantic features .3 Semantic Image Segmenation with Self-training Strategy .1 Panoptic Feature Pyramid Networks .2 Proposed Loss Function .3 Our Self-training Method .1 Datasets for Image Domain Translation.
Datasets for Semantic Image Segmentation.2 Nighttime Driving Testset.3 Extra Unlabeled Data Selecion. Day2Night Image Domain Translation Component.2 With Perceptual Loss Refinement Results .4 Semantic Image Segmentation Component .1 Daytime Cityscapes Images Training .2 Daytime and Nighttime Images Training .3 Daytime and Nighttime Images Training with Percep- tual Loss for Image Translation Module .4 Only-night Images Training with Perceptual Loss for Image Translation Module.5 Daytime and Nighttime Images Training with Focal Loss 73 4.6 Daytime and Nighttime Images Training with FID-based Method for Extra Unlabeled Data .3 Lessons from Series of Öxperiments.1 Improving GAN-based Method 80 5.2 Improving Semantic Image Segmentation Component. 80 A Publication 82 References 83 iv List of Figures 1.1 Illustration of the virtual vision of an autonomous car. Autonomous vehicles understand the surroundings with the help of cameras, sensors orradars, 60 .2 Artificial intelligence can well help automatically segment brain tumors.3 Satellite image segmentation helps automatically draw a map from an image.4 Definition of nighttime cityscapes image segmentation as a toy example.
Image segmentation model take a nighttime cityscapes image as input and process to output a segmentation map that assigns each pixel a semantic labgly. 4 21 Illustration of semantic segmentation problem. The Image Segmentation Model identifies that there exists a cat in the input image and which pixels belong to the cat.2 Results of semantic image segmentation and instance segmentation. Se- mantic segmentation on the left segments all the cats as one semantic class; instance segmentation on the right outputs each cat as a separate semantic segment.3 Fully Convolutional Networks [31] proposed by J.
Long et al in 2015.4 Upsampling example using transposed convolution. The input and kernel are 2 x 2 matrices, transposed convolution is performed with stride = 1. Multiplying the kernel with each component of the input, we get four 4x 4 matrices. The output is the sum of those four matrices.5 Fusing output to establish three variances of FCNs [31].6 Differences among the three variances of FCNs with an exemplary image from PASCAL VOC 2012 [31].7 Instances of the drawbacks of FCNs model [35].
(a) Too large objects face over segmentation issue; (b) Too small objects is missed.8 Deconvolutional Network [35] Architecture.9 Illustration of pooling and unpooling in DeconvNet [35].10 Comparison of the results of FCNs and DeconvNet [35].11 Visualization of decoder stage represented as heat maps. The information of the bicycle is gradually recovered in the last layers of DeconvNet [35].13 Extracted features are the results of a wider region |39].14 Architecture of Pyramid Scene Parsing Network |53]|.15 Results of Pyramid Scene Parsing Network on Cityscapes benchmark [53].16 Overall Architecture of RefineNet Model [2].17 Illustration of standard and atrous convolution. Atrous convolution gives a wider field of view with the same number of parameters compared with standard convolution.18 Illustration the differences when using standard and atrous convolution (r = 2). When applying standard convolution to the input image, we ownsize the image to H/2 x W/2 then apply filter 7 x 7.
In contrast, atrous convolution is directly applied to H x W input image. Basically, when using standard convolution, we process on 1/4 of the input image then upsize the feature map in output. Atrous convolution is applied to the whole image, therefore results in more dense feature maps.20 Atrous spatial pyramid pooling module in DeepLabv3 [4].21 Visualization of normal convolution.22 Visualization of separable convolution.23 Visualization of atrous separable convolution [ð].24 The application of GAN to generate datasets. New example images in datasets are generated by GAN.
(a) MNIST handwritten digit dataset, (b) CIFAR-10 small object photograph dataset, (c) Toronto Face database.25 Applications of image to image translation [16]. Image translation based on paired dataset such as day-to-night, sketch-to-image, segment map- to-photo and soon, 6. HH kg và 29 2.26 Applications of image to image translation [54]. Various domains are translated using unpaired image-to-image translation methods such as Monet-to-photo, zebras-to-horses, summer-to-winter and soon.27 An overview framework of GAN which contains the generator model G and the discriminator model D.28 Comparison result between traditional GAN and Conditional GAN [32].
(a) is randomly generated images, (b) is conditional generated images.1 Our Semantic Segmentation with Domain Adaptation Method Framework. The shared latent space assumption j30].3 Image-to-Image translation results in preliminary stage.4 Comparison of the result when the Perceptual Loss is applied and not.5 Overview of perceptual loss.6 General Panoptic Feature Pyramid Networks Architecture [20].7 Panoptic FPN Architecture for Semantic Segmentation.8 Illustration of self-training process in semantic segmentation module.1 Modified dataset contains two different domains which are daytime and nighttime distribufion.2 Exemplary images of Cityscapes Dataset.3 Exemplary images of Nighttime Driving Test.4 Results at 280k iterations. These images seem to be wrong located traffic and vehicle lights.5 Image-to-Image translation results with additional Perceptual loss.6 Image-to-Image translation results. It is too dark even for human vision.7 Image-to-Image translation results.
The results look too bright in com- parison with nighttime distribution.8 Examples of shiny sparkling of vehicle light in dataset.9 Examples of dark images in dataset.10 Examples of twilight images in đataset.11 Visualization of Segmentation Experiment-1 results.1 is the results of FPN-resnet101 trained on daytime images of Cityscapes; ID 1.2 is the results of self-training with the model from scratch; ID 1.3 is the results of self-training with checkpoint of ID 1. The unlabeled data in this experiment is 701 daytime cityscapes images of CamVid.12 Visualization of Segmentation Experiment-2 results.1 is the results of FPN-resnet101 trained on day and night images of Cityscapes; ID 2.2 is the results of self-training with checkpoint of ID 2. The unlabeled data in this experiment is around 15000 unlabeled nighttime cityscapes images of NEXĐT.13 Visualization of Segmentation Experiment-3 results.1 is the results of FPN-resnet 101 trained on day and images of Cityscapes; ID 3.2 is the results of self-training with the model with checkpoint of ID 3. The image translation module has been added perceptual loss to maintain objects features.14 Visualization of Segmentation Experiment-4 results.1 is the previous results trained on day and night images; ID 4.1 is the results of self- training with the model with checkpoint of ID 3.
only tests the effect of unlabeled data. Here we pick 1600 images from 15000 unlabeled nighttime images of NEXET dataset by histogram-based method.15 Visualization of Segmentation Experiment-5 results.1 is the results of FPN-resnet101 trained on only nighttime cityscapes images (generated by GAN); ID 5.2 is the model in ID 3.1 trained with more nighttime images; ID 5.4 are the results of self-training on 1600 unla- beled nighttime images selected by histogram-based method with the checkpoints of ID 5.16 Visualization of Segmentation Experiment-6 results.1 is the results of FPN-resnet101 trained on day and night cityscapes images with focal loss; ID 6.2 is the results of self-training on 1600 unlabeled nighttime images and also trained with focal loss. This experiment was held to compare the performance of focal loss and cross entropy loss among models.17 Visualization of Segmentation Experiment-7 results.1 is the results of self-training on around 1600 unlabeled nighttime images chosen by FID-based method and trained with cross entropy loss on checkpoint ID 3.2 is similar to ID 7.1 but with our proposed combined loss; ID 7.3 is to compared histogram-based with FID-based methods. FID-based method together with our proposed loss function yields the finest score.18 Visualization of Segmentation Experiment-8 results.1 is the results of self-training on around 1600 unlabeled nighttime images chosen by histogram-based method from checkpoint ID 5.2 is similar but unlabeled data is chosen with FID-based method, both 8.2 use cross entropy loss; ID 8.3 is the results of self-training on around 1600 unlabeled nighttime images from FID-based method with the checkpoint of ID 8.1 and the help of our proposed combined loss function.3 is our finest performance model.
TT viii List of 'Tables 21 A brief comparison of tasks in computer vision. 41 Cityscapes Dataset Class Definitions.2 Quantitative Result of Day2Night Translation Component.3 Results of Segmentation Experiment-1. Verifying self-training perfor- mance on daytime cityscapes dataset.4 Results of Segmentation Experiment-2. Narrowing down the distance between trainset and testset by adding generated nighttime cityscapes images together with self-training on true nighttime images.