UNIVERSITY OF CALIFORNIA, SAN DIEGO Spatio-Temporal Filtering For Image And Video Processing: Applications On Quality Enhancement, Coding and Data Pruning A dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Philosophy in Image and Signal Processing by Dũng Trung Võ Committee in charge: Professor Truong Q. Nguyen, Chair Professor Pamela C. Cosman Professor William S. Hodgkiss Professor Yoav Freund Professor Alon Orlitsky 2009 Copyright Dũng Trung Võ, 2009 All rights reserved.
The dissertation of Dũng Trung Võ is approved, and it is acceptable in quality and form for publi- cation on microfilm & electronically: Chair University of California, San Diego 2009 iii TABLE OF CONTENTS Signature Page. iii Table of Contents. iv List of Figures. vii List of Tables.
xi Vita and Publications. xiii Abstract of the Dissertation .1 Image and Video Processing Systems .3 Methods on Artifact Reduction .1 Spatio-Temporal Filtering for Quality Enhancement .2 Spatio-Temporal Filtering for Coding .3 Data Pruning-Based Compression. 14 2 Quality Enhancement for Motion JPEG using Temporal Redundancies .1 Translational Relation of DCT Coefficients .2 Quality Enhancement using Temporal Redundancies .3 Quality Enhancement for Real Video Sequences .4 Optimal Adaptive Filter for Arbitrary Number of Referenced Frames 24 2.1 Motion Vectors for Enhancement Process .2 Enhancement in the Ideal Case .3 Enhancement in Real Video Sequences. 37 3 Adaptive Fuzzy Filtering for Artifact Reduction in Compressed Images and Videos .2 Directional Fuzzy Spatial Filter .1 Directional Spread Parameter .2 Edge-based Directional Fuzzy Filter .3 Adaptive Fuzzy Compensated Spatio- Temporal Filter .4 Motion Compensated Metric for Flickering Artifact Evaluation .1 Enhancement for Compressed Images .2 Enhancement for Compressed Video Sequences.
61 4 Optimal Motion-Compensated Spatio-Temporal Filter for Quality En- hancement and Coding of H.264/AVC Video Sequences .1 In-loop Motion Compensated Spatio- Temporal Filter .2 Optimal Motion Compensated Spatio- Temporal Filter .3 Overlapped Motion Compensation .4 Optimal Weight for Inter-frame Coding .1 In-loop Motion Compensated Spatio-Temporal Filters .2 Optimal Motion Compensated Spatio-Temporal Linear Filter .3 Optimal Weight for Bi-predictive Coding. 83 5 Selective Data Pruning-Based Compression using High Order Edge-Directed Interpolation .1 Rate-Distortion Relation .2 Data Prune-Based Compression .3 Optimal Data Pruning .4 High Order Edge-Directed Interpolation .1 Single Frame-Based Interpolation .2 Multi-Frame-Based Interpolation .1 High Order Edge-Directed Interpolation .2 Data Pruning-Based Compression. 105 6 Conclusions and Future Works .1 Spatio-temporal Filtering for Quality Enhancement .2 Spatio-Temporal Filtering for Coding .3 Spatio-Temporal Filtering for Data Pruning .1 Spatio-temporal Filtering for Quality Enhancement .2 Spatio-temporal Filtering for Coding .3 Spatio-temporal Filtering for Data Pruning .4 Spatio-temporal Estimation for Video Processing. 113 vi LIST OF FIGURES Figure 1.1: Block diagram of an image and video processing system.2: An example of blocking artifacts for one zoomed-in part of 6th frame of Foreman sequence.3: An example of blocking artifacts over columns of 123rd row of Fig.4: An example of ringing artifacts in a synthesized image.5: An example of ringing artifacts over columns of 31st row of Fig.6: An example of ringing artifacts in a zoomed-in part of 6th frame of Mobile sequence.7: An example of flickering artifacts.8: Comparison between coding artifacts and Gaussian noise.9: The correlation between the current frame of compressed Mo- bile sequence and its surrounding frames.10: Block diagram of spatio-temporal filtering systems for quality enhancement.11: Block diagram of spatio-temporal filtering systems for coding.12: Block diagram of spatio-temporal filtering systems for data pruning.1: Translation between blocks of image xs and x.2: The original and linearized quantization functions.3: Block diagram of the enhancement algorithm.4: MSE for motion vectors (mb0 , nb0 ) = (0, 0) : (7, 7) and (mf0 , nf0 ) = (−7, −7) : (0, 0).5: Quality enhancement for ideal case - 6th frame of Mobile sequence.6: Quality enhancement for ideal case - 6th frame of Foreman sequence.7: Quality enhancement for City sequence.8: PSNR improvement for City frames compressed with quanti- zation matrix Q.9: PSNR for different options with integer pixel ME accuracy.10: PSNR for different options with half pixel ME accuracy.1: An example of directional JPEG artifacts with scaling factor of 4 for the quantization step matrix.2: Angle and spread parameter for directional fuzzy filter.3: Angles θ and θ0 of the edge-based directional fuzzy filter.4: Flow chart of the directional fuzzy filter.5: Block diagram of the adaptive fuzzy MCSTF.6: An example where the flickering metric in [4] has problem.7: Result of using a fuzzy filter.8: Pixel classification for directional filtering.9: Pixel classification for directional filtering.10: Comparison of filtered results.11: Zoomed images for comparison of filtered results.12: Comparison on the contribution of spatial and directional adap- tations.13: Zoomed images for comparison of Fig.14: Comparison of filter results for MJPEG sequences.15: Zoomed views for images in Fig.16: Comparison on PSNR of simulated methods for Mobile sequence.17: Comparison on flickering artifacts of simulated methods for Mobile sequence.18: Comparison of filter results for H.19: Comparison of PSNR for all frames in the Foreman sequence.20: Comparison of flickering metric for all frames in the Foreman sequence.21: Comparison of PSNR with different bit-rates of the Foreman sequence.1: Block diagram of the H.264/AVC encoder with in-loop MCSTF.2: In-loop coding and enhancement for GOP IBP.
The first row is the compressed sequence using conventional encoding scheme, the last rows is the compressed sequence using encoding scheme with in-loop enhancement and the middle rows explain step by step the encoding scheme with in-loop enhancement.3: Blocking artifacts of the motion compensated frames.4: Blocking artifacts in using cross-block and in-block cubics of MCSTF and MCTF.5: Overlapped blocks for motion compensation.6: Bi-predictive coding scheme.7: Comparison between conventional and proposed in-loop en- hancement H.8: Enhancement for 3rd frame of Foreman sequence.9: Zoom-ined part of Fig.10: PSNR and flickering artifact comparison for frame in Foreman sequences.11: Comparison in R-D curves for bi-predictive coding with differ- ent weight prediction option.12: Bi-predictive coding for 19th frame of Crew sequence.1: Block diagram of the data pruned-based compression.2: Block diagram of the data pruning phase.3: Data pruning for the 1st frame of Akiyo sequence.4: Block diagram of the single frame-based interpolation phase.5: Model parameters of sixth-order and eighth-order edge-directed interpolation.6: Block diagram of the proposed multi-frame-based interpolation for case of upsampling with ratio 1 × 2.7: Model parameters of 9th order edge-directed interpolation.8: Comparison of NEDI-6 and NEDI-9 to other methods.9: Comparison of NEDI-8 to other methods.10: Comparison results for R-D curves of single frame data pruning- based compression.11: Comparison of NEDI-6 to other interpolation methods in case of single frame data pruning-based compression.12: One zoomed in part of Fig.13: Comparison results for multi-frame data pruning-based com- pression.264/AVC compression and optimal data pruning-based compression with same bit-rate and PSNR values.264/AVC compression and optimal data pruning-based compression with same bit-rate and PSNR values. 105 ix LIST OF TABLES Table 2.1: PSNR enhancement in dB for ideal sequences.2: Comparison in PSNR improvement for different scenarios.1: Comparison of PSNR in units of dB for different methods.2: Percentage of the classified pixels.3: Comparison of PSNR in units of dB of Different Classified Pixels and of Spatial and Directional Adaptations.1: Operation of each step in Fig.2: PSNR and bitrate values for bi-predictive coding options. 99 x ACKNOWLEDGEMENTS First and foremost, I would like to express my deep admiration and true thanks to my advisor, Prof. His kindness makes me feel valued and comfortable during my studies at UC San Diego.
He motivates me every time I have a chance to discuss ideas or report progress with him. Undoubtedly, he inspires my research while still permits me a freedom in searching for new things. I also want to take this opportunity to thank the committee members: Prof. Yoav Freund and Prof.
Alon Orlitsky for their time and valuable comments. The suggestions of the committee members from my qualify exam substantially improved the quality of my thesis. Most of my knowledge on image processing comes from the course Digital Image Processing of Prof. Pamela Cosman in Fall 2005, and I would like to thank for her insightful lectures.
During the summer of 2007 I did my internship under the mentoring of Dr. Sehoon Yea and Dr. Anthony Vetro at Mitsubishi Electrics Research Laboratories. Their friendly and helpful support enriched my working experience and made me feel at home during the time I lived in Boston.
I still remember Dr. Anthony Vetro run quickly through the company aisles to save time. The work on data pruning-based compression would not be possible without the help of Dr Joel Solé and Dr Peng Yin at Thomson Corporate Research, where I spent the internship at Princeton in the summer of 2008. Their support during that time and when I came back UC San Diego encouraged me to keep working on this topic.
I would like send them great thanks for their approval of the topic and especially for the friendly discussions every Thursday. I would like to thank Prof. Jong-Ki Han, his student Chan-Won Seo and Daqian Jin from Motorola for their help in the project on video coding. This work was also encouraged by my professors and colleagues at my prior university, the Ho Chi Minh University of Technology.
I am grateful to Assoc. Thuong Le-Tien, Dr. Chien Hoang-Dinh and Assoc. Thanh Vu-Dinh for their understanding and support.
My labmates entertained and inspired me during the time I have studied at xi UC San Diego. I send my warm thanks to Cheolhong An, Min Li, Shay Harnoy, Ryan Prendergast, Stanley Chan, Vikas Ramachandra for their help and discus- sions. My motivation of quality enhancement lately comes from the beauty of the U. The most perfect images remain there forever and will always be the target for image processing.
Finally, I dedicate my dissertation to my parents, B ô´ và Me., for their un- conditional and unbounded love and also to my older sister, my brother-in-law and my younger sister, Chi., Anh, và Em gái, for their care of my parents during the last three and a half years. xii VITA 1980 Born, Dong Thap, Viet Nam. 2002 Bachelor, Electrical and Electronics, Ho Chi Minh City University of Technology, Viet Nam., Electrical and Electronics, Ho Chi Minh City University of Technology, Viet Nam. 2002-2005 Teaching Assistant, Center For Overseas Studies, Ho Chi Minh City University of Technology, Viet Nam.
2002-present Lecturer, Faculty of Electrical and Electronics Engineering, Ho Chi Minh City University of Technology, Viet Nam. 2005-2009 Research Assistant, University of California, San Diego. Summer 2007 Intern, Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA. Summer 2008 Intern, Thomson Corporate Research, Princeton, New Jersey.
Research Engineer, Digital Media Solutions Lab, Samsung Information Systems America Inc., Electrical Engineering (Image and Signal Processing), University of California, San Diego. PUBLICATIONS Dũng T. Võ, Chan-Won Seo, Daqian Jin, Jong-Ki Han and Truong Q. Nguyen, “Optimal Motion-Compensated Spatio-Temporal Filter for Quality Enhancement and Coding of H.264/AVC Video Sequences”, submitted to the IEEE Transactions on Image Processing, March 2009.
Võ, Joel Sole, Peng Yin, Cristina Gomila and Truong Q. Nguyen, “Selec- xiii tive Data Pruning-Based Compression using High Order Edge-Directed Interpola- tion”, submitted to the IEEE Transactions on Image Processing, January 2009. Võ, Truong Nguyen, Sehoon Yea, Anthony Vetro, “Adaptive Fuzzy Fil- tering For Artifact Reduction In Compressed Images And Videos”, accepted for publication in the IEEE Transactions on Image Processing, 2008. Võ, Truong Nguyen, “Quality Enhancement for Motion JPEG using Tem- poral Redundancies”, the IEEE Transactions on Circuits and Systems for Video Technology, vol.
Thuong Le-Tien, Chien Hoang Dinh, Dinh Viet Hao, Dũng T. Võ, “Neural Net- works - Based Equalizer Model Implemented To The DSP TMS320C6711”, Journal Of Science and Technology, Viet Nam, No. 40 + 41 /2003, ISBN 0868-3980, Viet Nam, 2003. Võ and Truong Nguyen, “Optimal Spatio-temporal Motion Compensated Filters for Quality Enhancement of H.264/AVC Compressed Sequences”, submitted to the 2009 IEEE Conference on Image Processing.
Chan, Dũng T. Võ and Truong Q. Nguyen, “Subpixel Motion Estima- tion For Translational Motions”, submitted to the 2009 IEEE Conference on Image Processing. Võ, Joel Sole, Peng Yin, Cristina Gomila and Truong Q.
Nguyen, “Data Pruning-Based Compression using High Order Edge-Directed Interpolation”, to appear in IEEE Conference on Acoustics, Speech and Signal Processing, Taiwan, 2009.