VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY FACULTY OF COMPUTER SCIENCE NGUYEN NGOC KHANH THESIS A MULTI-DEGRADATION APPROACH FOR DEEP LEARNING-BASED BACHELOR IN COMPUTER SCIENCE HO CHI MINH CITY, 2022 VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY FACULTY OF COMPUTER SCIENCE NGUYEN NGOC KHANH - 18520901 THESIS A MULTI-DEGRADATION APPROACH FOR DEEP LEARNING-BASED SINGLE IMAGE SUPER-RESOLUTION BACHELOR IN COMPUTER SCIENCE THESIS ADVISOR Dr. NGUYEN VINH TIEP HO CHI MINH CITY, 2022 DANH SÁCH HỘI ĐÒNG BẢO VỆ KHÓA LUẬN Hội đồng chấm khóa luận tốt nghiệp, thành lập theo Quyết định sé 36/QD-DHCNTT ngày 17/01/2022 của Hiệu trưởng Trường Dai học Công nghệ Thông tin. eee teeta een teense eeteeneeea eee ~ Ủy viên. Acknowledgements This thesis, a remarkable milestone in my study life, has been done throughout hard times and by incredible efforts.
I still remember the pressure I felt when I started this work. During this progress, I experienced all the feeling that a human being may know: struggling, disappointed, hopeless, then gradually a bit hopeful, successful, happy and proud. Obviously, I can never finish the thesis on my own. Fortunately, I still have my family, my friends, my advisors, to be with me on this long-term run.
First and for most, I want to send a thankful message to my family. They may contribute no knowledge, but theirs love and supports are incredibly valuable. Con cam on ba me va anh đã luôn ở bên cạnh con. Second, I would like to thank my advisor - Dr.
Nguyen Vinh Tiep, for his dedica- tion and motivation he has given to me. Without his knowledge and experiences, I will not be able to reach this milestone. He always keeps me on the right way, guide me to the final target. He is the lighthouse I believe.
Third, I would like to say I really appreciate my friends, the guys who have been with me since the day I started my university life and during my thesis progress. Particularly, my bros: Phuoc Hieu, Thanh Danh, Trung Nguyen, Angu, Vi Nghiem, Truong Phat and my best friend in UIT - Vien Duy. Especially, I value and respect Phuoc Hieu for his time and what he did teach me for over a year. Those guys do have extraordinary knowledge and skills.
They all changed my study life. Once again, thank you for everything. Time flies but I will never forget you and your supports. vi Contents Acknowledgements| xii m`=œM" oductio [1 Motvaton|.- [2 Challenge AY mrz.2 Prior Art in Single Image Super-Resolution|.3 Multiple Degradation Single Image Super-Resolution| .1 Non-blind Single Image Super-Resolution|.2 Blind Single Image Super-Resolution|.
3_ SR for Bicubic and Noisy degradation - SRBN 1 Problem Formulation|.1 Single Degradation Method|.2 Comprehensive Cross-Degradation Loss Function|. 26 {{_ SRUCD - SR Under Complex Degradation| 28 4.1 Constructing a Probabilistic Model for Our Proposed Method] .2 Structure of Our Proposed Probabilistic Model].2 Structural Similarity Index Measure (SSIM)|.2 Experimental Setup and Resultsl. 48 6_ Conclusion and Future Work| 52 6. eee 57 viii List of Figures 1.1 Medical images with different resolutions|.2 Comparision between interpolation methods|.2_ Architecture of VDSR modell.ẶẶ eee 12 25 EDSRarchitecturel.1 Difference between 2 types of LRimagel.2_ The proposed SRBN framework that solves two factors of degradation by online data augmentaton|.1 Our SRUCD model architecturel.2 The schematic illustration of FKP network}.3 Normalizing Flow lIlustration|.4 The archi! re of DnCNN network|.5 Comparison of DIP-FKP performance before and after attaching noise modeling modulel.6 The performance of DIP-FKP framework|.7 The training pipeline of our noise estimator|.1 Comparison between the results of our framework and the original eee 44 [5.2 Compare between non-ii.d versus iid noise image] .3 Qualitative result of ĐIPEKPmodelj.4 Qualitative result of VDNetmodel|.5 Qualitative result of our proposed model|.6 Qualitative result of our proposed modell.
51 List of Tables 5.1 SISR performance comparison on Set5, Set14 and BSD100 benchmark with scale factor 4Ì.2 Quantitative result of our proposed probabilistic model with different Noise Modelling module|.3 Results of an ablation study of our SRUCD model with different Noise [ Modelling modules|.- 50 xi List of Abbreviations ANN Artificial Neural Network CNN Convolutional Neural Network FC Fully-Connected GAN Generative Adversarial Network GPU Graphics Processing Unit HR High-Resolution ISR Image Super-Resolution LR Low-Resolution MSE Mean-Squared Error PSNR Peak Signal-to-Noise Ratio RNN Recurrent Neural Network SISR Single Image Super-Resolution SR Super-Resolution SSIM Structural Similarity Index Measure xii Abstract Single Image Super-Resolution (SISR) is a low-level task in Computer Vision aim- ing at reconstructing a high-resolution (HR) image from one low-resolution (LR) image. Technically, this task can be classified into three different approaches: interpolation- based, reconstruction-based and learning-based. Recent researches on image super- resolution have achieved significant performance thanks to the ability of deep learn- ing to extract image high-level features. However, the exists two challenges in the task of SISR.
First, the LR should be recovered with finer texture details at larger scales. Second, the predicted HR (a. SR) must satisfy the visual perception in order to be as natural as its ground-truth HR. In this research, we focus on deep- learning based approach to address the above problems.
This thesis details two proposed solutions using different approaches to recon- struct clean HR images from LR images that suffer from multiple degradations such as noise and blur. The first method uses an end-to-end framework to help tradi- tional SISR methods, i., those methods target at reconstructing HR images from spotless LR images, dealing with LR images containing Gaussian noise. We call this SRBN - SR for Bicubic and Noisy degradation. We introduce a Comprehen- sive Cross-degradation loss function to train this framework.
The second approach, SRUCD - SR Under Complex Degradation, which is constructed based on a proba- bilistic model. We use three separate modules to predict HR image, blur kernel and noise map from the given LR image. With this technique, our model can deal with LR images with multiple degradations. Chapter 1 Introduction In this chapter, we first introduce the problem of single image super-resolution, describing the motivation and the challenge of SISR task.
Then we briefly describe our solution and contributions. Finally, we outline the overall thesis structure.1 Motivation In the field of daily life, we are familiar with media such as image, video, audio and so on. Specifically, image is an indispensable part of our life thanks to its ability of capturing information of the real-world scenario without requiring too much on hardware. Only by owning a smartphone/ small camera, we are able to bring the surrounding real world into a 2D image that would be useful in certain cases.
How- ever, it was not in all cases that the captured image can satisfy our requirement. We may need a higher-resolution version of the image in order to see clearly the occlu- sion or detect better subjects in the scene. Furthermore, in specific majors, such as medical field or in outer space industry, a larger and clearer image would help treat and perceive the information better. Let us take some prime examples to be able to deeply understand howa high-resolution image plays an important role in some specific aspect of our life.1 Medical Field The need of super-resolution image in medical field, which ever since is known as directly related to human’s health-care responsibility, is high-demand.
In medi- cal department, there are medical imaging techniques such as CT or MRI that helps visualize the anatomy and the physiological processes of the body. An MRI scanner forms a strong magnetic field around the area of a subject to be imaged. Generally, an MRI image is in form of a gray-scale image and usually has low quality. A reso- lution for an MRI image is expected to be greater than 1mm, but to be able to reach that desire, its costs are low signal to noise ratio and longer time scan (5).
A higher- resolution would let experts give better decision on analysing victim’s problem and shorten their workload. Furthermore, with an enhanced image, many other com- puter vision tasks would be able to applied easier in medical images such as clas- sification or segmentation. Thus, it raises a motivation to find a method which can helps enhance the low-resolution MRI image to achieve desired version of image.2 Object detection Not at all the time that an image gives us good vision to detect all desired objects included in it. There would be occlusion in some cases or the object is at small spatial extent, such as vehicles in satellite imagery.
Therefore, enhancing a low-resolution image could give better performance in not only object detection problem but also many other tasks, as an increase in resolution should add more distinguishable fea- tures that an object detection algorithm can use for discrimination. A general thought could be made is using up-sample operation such as bi-cubic or nearest neighbor interpolation, but these methods only return a larger scale of the Chapter 1. Introduction (A) Human brain horizontal slice at the resolu- (B) Human brain horizontal slice at the resolu- tion of Imm tion of 0.7mm (C) Human brain longitudinal slice at the reso- (D) Human brain longitudinal slice at the reso- lution of Imm lution of 0.1: Medical images with different resolutions Chapter 1. Introduction 4 image with noise, blurry details and artifacts, that neither better perception nor use- ful information.
Whereas we want the image bigger in size but still keep at least the same information with the original one. Many modern cameras nowadays are able to capture image with the resolution up to 4K or higher, but they are usually not af- fordable for individual demand. Therefore, with the urgent need for super-resolution image, techniques for recover HR image are requisite and are the most realistic ap- proach.2: Comparision between interpolation methods for image up- sampling. Using linear interpolation results in blurry image, while NN interpolation creates discrete boundaries between pixels Chapter 1.2 Challenges Despite these clear advantages of SR image, the progress of reconstructing SR image from a LR one is still a struggling problem.
As we mentioned in Sec. we not only need to predict a high-resolution version of the image but also need to guarantee the structure of the original image is available in the predicted one. However, recent works in SISR task, which will be discussed carefully in the next chapter, are not able to totally recover fine details texture. Particularly, they put the problem under an ideal case that the input LR images are smaller version of the HR images, while in practice these images could be affected by noises and blurry arti- facts.
Therefore, the traditional DL methods for SISR task may fail in reconstructing LR images in real-world setting.3 Contributions Because of the difficulties mentioned above, we figure out that we need flexible solutions for the SISR problem. The word "flexible solutions" means those methods should be able to reconstruct well HR images from LR images under any circum- stances, whether they contain artifacts or not. This thesis proposes a new framework that helps traditional SISR methods deal with more complicated LR images, i., those contains artifacts. Besides, we intro- duce a new probabilistic SISR model to reconstruct LR images containing noise and blurry patterns.
Briefly saying, our contributions are: ¢ Anend-to-end deep learning framework that helps extend the ability of tradi- tional SISR model. ¢ Anew loss function helps train the proposed framework. Introduction 7 ¢ A probabilistic model which helps reconstruct HR images from LR images with multiple degradation.4 Disposition This introductory chapter served as an introduction that gives the motivation, encountered difficulties in the field of SISR, as well as the contribution of this re- search. Chapter|2]will give an overall review of recent SISR algorithms and carefully describe their strength and weak points.
Chapter Bland Chapter ] provide details about our 2 proposed model in this thesis. Next, Chapterl]brings about quantitative and qualitative results of our introduced models. Lastly, Chapter|6|summarizes the work and its limitations with further discussion about future work.