LÊ QUỐC ANH POINT CLOUD COMPRESSION FOR HUMANS AND MACHINES LUẬN VĂN THẠC SĨ NGÀNH ĐIỆN TỬ, NĂNG LƯỢNG ĐIỆN, TỰ ĐỘNG HÓA CHUYÊN NGÀNH KỸ THUẬT TRUYỀN THÔNG VÀ DỮ LIỆU NGƯỜI HƯỚNG DẪN KHOA HỌC TS. GIUSEPPE VALENZISE TS. LÊ VŨ HÀ HÀ NỘI, NĂM 2024 POINT CLOUD COMPRESSION FOR HUMANS AND MACHINES Master thesis of Paris-Saclay University and VNU University of Engineering and Technology Specialization: M2 Data and Communication Engineering Research unit: Advanced Institute of Engineering and Technology, VNU University of Engineering and Technology Thesis presented at Hanoi, on 29 January 2024 LE Quoc Anh Committee Arnaud BOURNEL Paris-Saclay University Chairman Pierre DUHAMEL CNRS, CentraleSupelec, Paris-Saclay University Rapporteur NGUYEN Linh Trung VNU University of Engineering and Technology Examiner Giuseppe VALENZISE CNRS, CentraleSupelec, Paris-Saclay University Examiner LE Vu Ha VNU University of Engineering and Technology Examiner Thesis Supervision Master Thesis Giuseppe VALENZISE CNRS, CentraleSupelec, Paris-Saclay University Supervisor. LE Vu Ha VNU University of Engineering and Technology Co-supervisor.
Acknowledgements While pursuing the Master’s degree, especially during the thesis period, although many things did not go as expected, I am very grateful for the opportunity to be part of such an amazing learning and research environment. During this time, I had the chance to engage and collaborate with inspiring professors, colleagues, and friends who greatly contributed to my academic journey, and I appreciate their support. First, I would like to express my gratitude to my supervisors, Prof. Giuseppe Valenzise, Prof.
Chaker Larabi, and Prof. Le Vu Ha, for their continuous support and guidance. Without their support, this thesis would not have been possible. I want to thank the coordinators of this Master’s program, Prof.
Pierre Duhamel, Prof. Nguyen Linh Trung, and Dr. Nguyen Hong Thinh, for their continuous efforts in operating the program and providing me with the best study environment. I would also like to thank the program’s lecturers for delivering valuable and enriching lessons during my Master’s studies.
My gratitude goes to the Advanced Institute of Engineering and Technology (AVITECH) members. Special thanks to the institute’s director, Prof. Nguyen Linh Trung, for supporting me in every aspect and ensuring the best working environment during my Master’s internship in Vietnam. I want to express my appreciation to my labmates, Vu Duy Thanh, Do Hai Son, Tran Trong Duy, Doan Quang Manh, Le Cong Hieu, and other students, for providing advice, engaging in discussions, and sharing hardware resources for my research.
I would like to thank Dr. Luu Manh Ha for giving me valuable research experience. I want to thank Pham Xuan Loc for collaborating and discussing research problems. Many thanks to Hoang Van Nhat for supporting me during the application process for this program.
I want to thank the Master, PhD Scholarship Programme of Vingroup Innovation Foundation (VINIF) for funding me to pursue this Master’s program. Finally, I am grateful for the endless encouragement and support from my family. i Authorship I solemnly declare that my thesis, titled "Point Cloud Compression for Humans And Machines", is my own research work conducted under the guidance of Prof. Giuseppe Valenzise and Prof.
Le Vu Ha. The sources used in the thesis are explicitly mentioned in the reference section, with proper citations. The data and results presented in the thesis are entirely truthful, and there is no copying from the works of others. If any discrepancies are found, I take full responsibility and am subject to any disciplinary actions imposed by the university., 2024 Student Le Quoc Anh 1 Contents 1 Introduction 6 1.2 Motivation, objective and contribution.
9 2 Background and state-of-the-art 11 2.1 Point Cloud Compression .2 Learning-base Approaches .2 Point Cloud Classification .3 Coding for Humans and Machines. 21 3 Proposed residual method in scalable point cloud coding for humans and machines 23 3.2 Synthesis transform for latent space representation of base branch (hr (.2 The architecture of analysis and synthesis transform of the enhancement branch .4 Entropy rate modeling. 28 4 Experiments and results 29 4.1 Point cloud compression for classification .2 Point cloud compression for humans. 36 5 Conclusions and future works 40 5.
40 3 List of Figures 2. The classification network takes a sparse tensor as input.1 Overall architecture of residual method in scalable point cloud coding for humans and machines. The dashed line indicates that the enhancement network has no influence on the base network.2 Proposed codec for dense point cloud compression for classification. The input of the codec is the latent space of the PCGCv2.3 The process of transforming the feature vector Zb for applying in the residual method using the synthesis transform block hr (.4 The architecture of the analysis and synthesis transform of the enhancement branch is built upon the Residual Bottleneck Block (RBB) as the fundamental unit.1 Example point cloud data from ModelNet10 dataset with three resolutions (r): (a) 64, (b) 128, and (c) 256.2 Rate-accuracy curves evaluated on the ModelNet10 dataset voxelized with three resolutions (r): (a) 64, (b) 128, and (c) 256.
P is the number of points in the input X. "*" denotes a dense point cloud.3 Rate-distortion curves of ModelNet10 dataset: (left) D1-based PSNR, (right) D2-based PSNR with three resolutions (r) (a) 64, (b) 128, and (c) 256. 37 4 List of Tables 4.1 The statistic information of the point cloud data in the selected ModelNet10 dataset.2 BD rate and maximum accuracies per codec evaluated on ModelNet10 voxelized with three resolutions. P is the number of points in the input X.
"*" denotes dense point cloud.3 BD-rate gains on ModelNet10 of the proposed codec against other compression methods using D1 and D2 based BD-rate measurement.4 Average coding time of different methods on the ModelNet10 dataset.1 Context In recent years, technologies related to Extended Reality (XR)1 have developed enormously, leaving a significant impact on many different aspects of our daily lives [1]. The continuous development of XR is partly based on the advance of many capturing techniques for three-dimensional (3D) data. Among the types of data used for XR, point clouds are a common and widely used data type for 3D object representation and scenes [2]. Point clouds are applied in many applications, such as VR [3], MR [4], construction [5], cultural heritage [6], etc.
A point cloud consists of a set of discrete points containing coordinates information (x, y, z) and additional attributes such as color, normals, reflectance, etc. In addition, depending on whether a point cloud includes a temporal dimension, we can classify the point cloud data into static and dynamic forms. In most applications, the number of points in realistic and immersive point clouds easily ranges in the order of millions, and the attributes are also complex. Therefore, if we transmit uncompressed point cloud data, it will exceed the bandwidth of most current communication systems.
For example, streaming a dynamic point cloud without compression requires a bandwidth of 3. Thus, effective and efficient point cloud compression methods are essential to enable the use of point clouds in practical applications. Due to the substantial increase in 3D point cloud data and the demand for an efficient compression 1 We use XR as an umbrella term to include Virtual Reality (VR), Mixed Reality (MR), Augmented Reality (AR), and other related technologies. 6 method for immersive applications, point cloud compression has become the object of standardization by the Moving Picture Experts Group (MPEG).
MPEG has proposed two standards for point cloud compression, which are Geometry-based Point Cloud Compression (G-PCC) and Video-based Point Cloud Compression (V-PCC) [7]. G-PCC and V-PCC are based on traditional approaches, such as octree decomposition or triangulated surface model for G-PCC and 3D-2D projection for V-PCC [2]. Recently, powerful deep learning-based image/video compression methods have been published. These image/video compression methods show promising results compared to traditional methods such as JPEG2000 and have comparable results with the advanced method currently HEVC [8, 9].
Taking advantage of the idea of using neural networks to extract many compact latent spaces to improve compression efficiency, many point cloud compression methods based on deep learning have been published and show promising results compared to the compression standards issued by MPEG [10, 11]. Conventionally, point cloud compression methods are optimized to reconstruct point clouds for human visualization. However, in the digital era, point clouds are also used to provide information for machines to serve tasks such as scene understanding, self-localization, tracking, pose estimation, and many others [12]. Therefore, point cloud compression methods must be improved or redesigned to maximize performance for machine tasks.
In addition, we also need to pay attention to “human in the loop” scenarios, where humans also need to observe visual content.2 Motivation, objective and contribution 1.1 Motivation Point cloud classification plays a pivotal role in a wide range of applications, including surveillance, autonomous driving, environmental monitoring, and augmented reality [13]. This process enables the identification and labeling of point cloud objects and structures. In surveillance applications, point cloud classification aids in detecting objects of interest, enhancing security measures [14]. In autonomous driving systems, point cloud classification contributes to scene understanding, allowing vehicles to recognize and respond to different objects on the road [15].
The versatility of point cloud classification extends to environmental monitoring, where it assists in analyzing terrain, vegetation, and other features [16]. 7 Moreover, in augmented reality experiences, accurate classification ensures realistic virtual object placement and interaction with the real-world environment [17]. In real-world applications with limited transmission bandwidth and storage capacity, it is essential to compress point clouds at a specific rate/quality that meets the requirements of the particular application. In the context of point cloud compression, achieving precise reconstruction of geometric information is paramount [18].
This precision is essential to enable high-quality rendering and support interactive applications. Although recent efforts have investigated lossless point cloud geometry compression, even with state-of-the-art methods, the compression efficiency struggles at around 1 bit per occupied voxel [19]. This limitation results in substantial storage and transmission costs for large point clouds. Therefore, using lossy compression methods will yield better performance, especially when providing specific rate/quality values for each particular application.
This helps reduce storage and transmission costs for large point clouds. Dense point clouds are needed for human visualization to achieve a satisfactory rendering quality [20]. However, most studies on machine vision tasks, such as point cloud classification, use sparse point clouds [21, 22]. This is because sparse point clouds have enough information to achieve high performance for machine vision tasks and significantly reduce computation costs.
The Joint Photographic Experts Group (JPEG) has recently acknowledged the potential of deep learning- based coding. In response to this recognition, the JPEG Pleno PCC projects have been initiated with the goal of establishing specific coding standards for point clouds using deep learning techniques [23]. The objective of the JPEG Pleno PCC project is to develop a learning-based coding standard for point clouds and their associated attributes. This standard aims to provide a single bitstream, compact compressed domain representation that supports advanced and flexible data access functionalities.
The scope includes human visualization, aiming for competitive compression efficiency compared to state-of-the-art point cloud coding solutions. Simultaneously, it aims to deliver effective performance for 3D processing and machine vision tasks. In collaborative scenarios [24], where edge devices send data to the cloud for processing, transmitting only the essential information for tasks is efficient. Generating distinct representations for each subset of tasks becomes impractical with an increasing number of tasks.
Additionally, if information for some tasks 8 has already been sent, and a broader set of tasks is needed for the same input, sending a new representation would involve redundant information [25]. Thus, organizing task information in a scalable fashion should be an efficient way [26], sharing base representations among tasks and needing only extra information for more specific tasks.2 Objective Building upon the motivations outlined above, this thesis explores a learning-based coding solution for static 3D point clouds, aiming to generate representations suitable for both machine vision tasks and human visualization. Our initial focus is on investigating dense point cloud geometry data.