Hanoi University of Science and Technology School of Information and Communication Technology Master Thesis in Data Science Contrastive learning for semantic segmentation in endoscopy NGUYEN VIET HOAI hoai.vn Supervisor: Dr. Dinh Viet Sang Hanoi 10-2023 THESIS ASSIGNMENT 1. Student’s information : Name : Nguyen Viet Hoai Phone : 0946071194 Email: hoai.vn Class : Data Science (Elitech) Affiliation : Hanoi University of Science and Technology Duration : 10/2021 - 10/2023 2. Thesis title : Contrastive learning for semantic segmentation in endoscopy 3.
Declarations/Disclosures : I, Nguyen Viet Hoai, confirm that this Master’s Thesis is the product of my original research conducted under the supervision of Dr. Dinh Viet Sang. The findings presented in this thesis are a true representation of my own work and have not been plagiarized from any other sources. All the sources referred to in this Master’s Thesis, including images, tables, graphs, and quotations, have been appropriately cited and credited.
This thesis has not been previously submitted for assessment or accreditation by any other educational institu- tion. I take full responsibility for any potential violations found within this thesis. Hanoi, date 13 month 10 year 2023 Author Nguyen Viet Hoai 4. Attestation of thesis advisor : I confirm that I have thoroughly reviewed this thesis, and in my judgment, it meets the necessary standards in terms of scope and quality to qualify for the Master of Science degree.
Hanoi, date 13 month 10 year 2023 Thesis Advisor Dr. Dinh Viet Sang 2 Abstract Semantic segmentation is a fundamental task in computer vision, which aims to distinguish pixels of images from corresponding semantic classes. Semantic seg- mentation has many applications such as automatic-driven, facial recognition, and medical imaging. Machine learning image segmentation plays a critical role in de- tecting colon polyps that appear in endoscopic images.
Colon polyp recognition is challenging for the automatic analysis of endoscopic images in computer-aided clinical support systems. In polyp segmentation, recent deep learning models have shown impressive results, they often reveal weaknesses when dealing with scarce and imbalanced training data. To overcome this issue, we present a novel effec- tive method for polyp semantic segmentation using contrastive learning supported by a memory bank. Contrastive learning disentangles the feature representation of different object classes.
Meanwhile, the memory bank gathers hard pixel samples within successive mini-batches to deal with the scarcity of polyp class that may occur in a single mini-batch. Furthermore, we leverage the teacher-student train- ing to force the model to learn consistent feature representation, resulting in final polyp segmentation improvement. Experiments show that our method yields supe- rior performance by a large margin compared to other state-of-the-art approaches. Our code is available at: https://github.com/hoainv99/meco-polyp.
Contents Contents 1 List of Figures 3 List of Tables 4 1 Introduction 6 1.4 Outline of the thesis .1 Deep learning and Medical segmentation .1 Image Classification .1 Convolutional Neural Networks .3 Medical image segmentation .2 Types of learning .2 Semi-supervised learning .4 Contrastive Learning for Semantic Segmentation .2 Global Context-Aware Modules .3 Active Sampling and Memory Bank .1 Active Key Sampling.2 Hard Query Sampling in Memory Bank. 33 4 Experiments and Evaluation 35 4.3 Training Setup and Strategies .2 Dice coefficient .3 Intersection over Union. 41 5 Conclusion 44 Bibliography 45 2 List of Figures 2.1 The transformer architecture [1] .2 The Vision Transformer architecture [2] .1 Overall architecture of our MeCo framework includes supervised, con- sistency, and MeCo losses. The supervised loss is computed based on the original data.
The consistency loss is computed using Cutmix and the teacher model to generate augmented data and corresponding pseudo labels. The MeCo loss is computed based on feature embed- dings taken from a memory bank.2 Our proposed model architecture .3 The architecture of baseline model [3] .4 The architecture of SegFormer [4] .5 Grouped compact generalized non-local (CGNL) module [5].6 A Squeeze-and-Excitation block [6] .7 Feature selection module [7] .8 Feature alignment module [7] .1 Intersection over Union based on threshold .2 Our method’s true and false results in Neo-large test. An easily ob- servable limitation is that the model struggles to identify small polyp regions accurately.3 Ablation studies on the the NeoPolyp-Small dataset .4 The results of challenging images in the Kvasir dataset are compared among different models.5 The results of challenging images in NeoPolyp-Large datasets are compared among different models. 43 3 List of Tables 4.1 Performance metrics of different models on five binary polyp datasets.2 Performance metrics of different models on NeoPolyp-Large dataset.3 Performance mDice of different models on NeoPolyp-Small.
40 4 List of abbreviations CNN Convolutional Neural Network FC Fully-Connected IOU Intersection Over Union FCN Fully Nonvolutional Network CRF Conditional Random Fields EMA Exponential Moving Average TP True Positive FP False Positive FN False Negative CR Consistency Regularization MB Memory Bank Chapter 1 Introduction 1.1 Overview Colorectal cancer (CRC) is a type of cancer characterized by uncontrolled cell growth in the colon. CRC is the third most common type of cancer around the world [8]. Therefore, preventing CRC by screening tests and removing preneoplastic lesions (colorectal adenomas) is critical and has become a worldwide public health priority. Colonoscopy is an effective technique for CRC screening and prevention since it can provide location and appearance information of colorectal polyps, enabling doctors to remove these before they develop into CRC.
A number of studies have shown that early colonoscopy has contributed to a 30% decline in the incidence of CRC [9]. In addition, colon polyps are abnormal growths that occur in the lining of the colon or rectum. Although most polyps are benign, some can develop into cancer over time. Early detection and removal of polyps are crucial in preventing colorectal cancer, the second leading cause of cancer-related deaths.
Regular screening tests such as colonoscopies enable the identification of polyps at an early stage when they are easier to remove. Individuals significantly reduce their risk of developing colorectal cancer by undergoing timely polyp removal. Semantic segmentation is a crucial computer vision task that splits an image into meaningful segments based on semantic content. In other words, it assigns a label to each pixel to represent the object or region to which it belongs.
This contrasts the object detection problem, which identifies objects in an image and draws bounding boxes around them. Semantic segmentation provides a more detailed understanding of the content in the image. It is commonly used in various applications, such as image understanding, scene analysis, automated driving, medical image analysis, etc. It is a critical component in tasks that require pixel-level analysis and is the underlying technology for computer vision and artificial intelligence.
Polyp segmentation is a particular semantic segmentation task and is essential in screening procedures for early detection of colorectal cancer. By precisely segment- 6 ing polyps, clinicians can make informed decisions regarding diagnosis, treatment, and intervention strategies, ultimately improving patient outcomes. Utilizing AI models to segment polyp regions is an essential and challenging task, as it plays a crucial role in improving support and accuracy in screening tests. This task is nec- essary for early detecting polyps, which helps doctors diagnose colorectal polyps in endoscopy images.
Polyp segmentation is a challenging task; almost recent models have trouble with the scarcity of data and the difficulty in segmenting polyp regions when polyps vary in size and color. Among the diverse techniques employed for polyp segmentation, early learning- based methods heavily depend on manually designed features [10, 11], such as color, texture, shape, appearance, or a combination thereof. Typically, these approaches involve training a classifier to determine polyps from their surrounding context. However, these models frequently encounter a notable issue regarding their ability to detect polyps accurately.
The primary explanation lies in the limited capacity of hand-crafted features to adequately address the substantial variations within the polyp class and the minimal differences between polyps and challenging imitating elements [12]. Recently, there has been a surge in the development of deep learning- based approaches for polyp segmentation [12, 13]. While these methods have shown progress, they primarily rely on identifying polyps using bounding boxes, which re- sults in imprecise delineation of polyp boundaries. In response to this challenge, Brandao et al.
[14] introduced a Fully Convolutional Network (FCN) with a pre- trained model to identify and segment polyps accurately. Akbari et al. [15] further enhanced polyp segmentation accuracy by employing a customized version of FCN. Drawing inspiration from the successful application of U-Net [16] in biomedical im- age segmentation, U-Net++ [17] and ResUNet++ [18] were harnessed for polyp segmentation, yielding promising results in segmenting the entire polyp area.
How- ever, these methods overlook the crucial area-boundary relationship, which is vital for improving segmentation performance. To address this shortcoming, psi-Net [19] integrated area and boundary information into polyp segmentation. Nevertheless, the intricate connection between the area and the boundary was not fully captured. Fang et al.
[20] proposed a three-step selective feature aggregation network in- corporating area and boundary constraints for polyp segmentation. This method explicitly considers the interdependency between areas and boundaries, yielding fa- vorable results with the inclusion of edge supervision. However, it should be noted that this approach is computationally intensive, requiring more than 20 hours for processing, and is susceptible to overfitting. These models demonstrate impressive results when a lot of data is available.
However, these approaches are considerably less effective due to two significant reasons. First, the polyp datasets are scarce and 7 imbalanced. Second, in colonoscopy images, the boundary between a polyp and its surrounding mucosa is usually blurred and lacks the intense contrast required for segmentation approaches. To address these issues, we propose a novel method leveraging consistency reg- ularization and contrastive learning to build a strong and view-invariant feature representation.
Moreover, we utilize a memory bank to store hard pixel embeddings generated by a so-called active sampling strategy. The usage of hard pixel embed- dings from the memory bank reduces the effect of data imbalance between polyp and background classes since the polyp pixels can be accumulated across successive mini-batches. With regard to the use of contrastive learning, we can increase the accuracy of the model for detecting polyp’s surrounding mucosa. Through experimental evaluations on challenging polyp and neo-polyp datasets, we demonstrate the superiority of our approach over state-of-the-art methods.
Our method achieves significant improvements in mean IOU and mean Dice scores, in- dicating the accuracy of our polyp segmentation.2 Objectives The objective of this study is to effectively apply contrastive learning and some semi-supervised techniques to the baseline model to improve accuracy. Consider- ing specific datasets, the model demonstrates the superiority of our approach over state-of-the-art methods. This work opens up a new research direction in the iden- tification of colorectal polyps in endoscopic images, thereby enhancing the accuracy and performance of current models.3 Main contributions In summary, the main contributions of this paper are as follows: • We propose a novel method that leverages contrastive learning supported by a memory bank to enhance feature representation. The memory bank accumu- lates hard pixel samples within successive mini-batches to address the scarcity of polyp pixels in a single mini-batch of training images.
• Inspired by semi-supervised learning, we utilize teacher-student training to force the model to learn a robust feature representation that leads to improve- ment in final segmentation results. • Extensive experiments on many popular benchmark datasets show that our method consistently outperforms many other state-of-the-art approaches.4 Outline of the thesis The rest of this paper is organized as follows: • Chapter 2, we briefly survey related work. • Chapter 3, we describes our proposed method. • Chapter 4, we present and evaluate the results of the experiments.
In addition, we show the effectiveness of each of our proposal components. • Chapter 5, we conclude our contribution includes our proposal method and our experiments. Finally, we show future work. 9 Chapter 2 Related Work 2.1 Deep learning and Medical segmentation In today’s world, Deep learning is increasingly finding widespread applications, spanning diverse fields including computer vision, natural language processing, and speech recognition.