VIETNAM NATIONAL UNIVERSITY, HO CHI MINH CITY HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY FACULTY OF COMPUTER SCIENCE AND ENGINEERING REPORT CAPSTONE PROJECT 3D MULTI-OBJECT TRACKING IN LIDAR POINT CLOUD MAJOR: COMPUTER SCIENCE THESIS COMMITTEE : COMPUTER SCIENCE - CLC 02 SUPERVISOR(s) : LE HONG TRANG, Assoc. REVIEWER : NGUYEN DUC DUNG, Dr. ————o0o———– STUDENT : DAM NGOC HIEU - 2052469 HO CHI MINH CITY, December 2024 Supervisor’s Signature Date Assoc. Le Hong Trang, Dr (Supervisor) Associate Professor Faculty of Computer Science and Engineering i Declaration Of Authenticity I am Dam Ngoc Hieu, a student of class CC20KHM2, at the Faculty of Com- puter Science and Engineering, Vietnam National University - Ho Chi Minh City University Of Technology.
I hereby affirm that the thesis "Development of a Model for 3D Multi-Object Tracking" is my own research, study, and development under the guidance of Assoc. Le Hong Trang. It is not a copy from other documents or research works without proper citation in the references. I take full responsibility for this declaration.
Ho Chi Minh City, December 2024 Author, Dam Ngoc Hieu i Acknowledgement To complete this thesis, I would like to express my gratitude to all the es- teemed professors and lecturers at Ho Chi Minh University of Technology for their dedicated teaching and imparting valuable knowledge to me throughout my academic journey. Above all, I am deeply grateful to my supervisor, As- soc. Le Hong Trang, for his dedicated instruction, teachings and his enthusiastic guidance and advice throughout my research, development, and completion of this thesis. ii Abstract This thesis focuses on proposing a new model for advanced 3D multi-object tracking based on a baseline.
The algorithm uses LiDAR point clouds to im- prove the autonomy of driving systems. Research highlights in detecting and tracking objects such as cars, cyclists, and pedestrians, emphasizing high pre- cision and capability blockage handling. iii Contents 1 Introduction 1 1.4 Convolutional Neural Networks .1 Intersection of Union .2 Multiple Object Tracking Accuracy .3 Average Multiple Object Tracking Accuracy .4 Scaled Average Multiple Object Tracking Accuracy .5 Multiple Object Tracking Precision .6 Average Multiple Object Tracking Precision .6 PointRCNN: 3D Object Detection with Point Clouds .1 Overview of PointRCNN .2 Point-wise Feature Extraction .3 Region Proposal Network .5 End-to-End Training and Loss Functions .6 Advantages of PointRCNN .7 Applications and Use Cases .8 PointRCNN Output and Integration with 3D MOT .9 Challenges and Future Directions .7 Particle Filter-Based Tracking .1 State Representation with Particles .6 Advantages of Particle Filter-Based Tracking .8 Particle Filter-Based Tracking .1 State Representation in Kalman Filter .3 Advantages of Kalman Filter-Based Tracking .9 Comparison of Kalman Filter and Particle Filter .1 FANTrack: 3D Multi-Object Tracking with Feature Associa- tion Network .2 Robust Multi-Modality Multi-Object Tracking .3 3D Multi-Object Tracking: A Baseline and New Evaluation Metrics .4 EagerMOT: 3D Multi-Object Tracking via Sensor Fusion .1 Detection Data Preparation .2 Handling Frame-specific Data .3 Setting Up Directories and Paths .5 Tracking and Saving Results .6 Post-Processing and Result Integration .4 Tracking Algorithms in 3D MOT .1 Tracking-by-Detection .1 Detection Output Association .4 Track Creation and Deletion. 56 References 57 vii List of Figures Figure 1.1 An example of tracking by LiDAR[2] .3 Convolutional Neural Networks architecture[5] .1 Overall architecture of the FANTrack model[7] .2 Overall architecture of the AB3DMOT [12] .1 Architecture of proposed model.
45 viii List of Tables Table 4.1 3D MOT Models: Performance Metrics Overview about Car .2 Car Evaluation Summary .3 Cyclist Evaluation Summary .4 Pedestrian Evaluation Summary. 52 ix Chapter 1 Introduction Chapter 1 provides an overview of the research project, outlining its objectives and goals. Additionally, it presents the structure of the report.1 Motivation In the modern world of fast-developing technologies, the ability to track and an- alyze the motion of several objects in 3D space has turned into a very important feature. The topic "3D multi-object tracking in LiDAR Point Cloud"[1] is not so unfamiliar any more.
It goes into advanced methods and technologies that are supposed to allow precise tracking of multiple objects using only LiDAR point clouds, which are datasets created by LiDAR (Light Detection and Ranging) sensors.1: An example of tracking by LiDAR[2] 1 LiDAR has evolved into one of the most core and highly visible technologies since its introduction into 3D object tracking, as it fills many critical market needs for highly resolved, spatially accurate data. LiDAR works by firing laser beams at targets and determining how long those beams take to bounce back from objects. The data from these laser beams then contribute to creating dense 3D point clouds that represent the surrounding environment in great detail. In- deed, such point clouds are a key component in the identification and tracking of vehicles, pedestrians, and obstacles in dynamic driving scenarios.
The goal of this research is to improve the quality of object tracking based on LiDAR point clouds, which is an important contributor to improved safety, operational effi- ciency, and adaptability in dynamic environments within the current and future autonomous vehicle and transportation systems.2 Goals The core goal of this research is to develop and refine 3D multi-object tracking algorithms using only LiDAR point clouds. First, there is a need for efficient processing and filtering of LiDAR data to ensure its accuracy for analysis. Li- DAR point cloud data often contains noise, gaps, or anomalies that need to be addressed before it can effectively be utilized for object tracking. The other main objective is to come up with object detection algorithms strong enough to accurately outline and keep track of all kinds of objects using 3D point cloud data.
Most of the problems in this aspect will be the differentiation between different objects based on shape, size, and pattern of motion. Of importance will be the development of an algorithm that can track multiple objects simultaneously. The development of the algorithm will be important in considering a number of challenges in tracking multi-objects that move through dynamic environments; for example, practical applications include congestion 2 traffic or making decisions in real time. The tracked systems will also need to detect not only the current status of an object, but also the future movements, considering past trajectories—a very vital issue related to several real-time ap- plications in self-driving cars.
The study also ensures that the system performs well in real time. The au- tonomous vehicle and traffic management systems require ultra-low latency re- sponses to perform efficient traffic management on the road. Therefore, this research will focus on optimizing the computational efficiency of tracking al- gorithms, including developing parallel processing techniques, to utilize high- performance computing resources for fast and accurate processing of huge vol- umes of LiDAR data. The final step of the research will involve the evaluation of the proposed tracking algorithms in terms of performance on benchmark LiDAR datasets.
Accuracy, robustness, and speed in processing will be under consideration, toward realiz- ing a system that would easily be incorporated into autonomous driving. Such a system is envisioned to give very high returns in improving vehicle safety, navigation, and decision-making within complicated environments.3 Scope This research is about 3D multi-object motion tracking of objects in dynamic environments using LiDAR point cloud. The main subsection of the project in- volves the development of algorithms that can detect, track, At the same time, it is possible to predict the movements of many objects, such as vehicles and pedestrians, using only data collected by the LiDAR sensor. The research scope of this thesis includes the entire LiDAR data processing process, starting from data collection and preprocessing.
LiDAR sensors, after 3 using laser beams, can create point clouds that represent the 3D structure of the environment, but these point clouds often require significant preprocessing to remove noise, fill gaps, and improve accuracy. This research will initially focus on methods to clean and segment LiDAR point clouds, allowing the system to accurately identify individual objects in the environment. Once the point clouds are processed, the next step is object detection and clas- sification. The ability to identify and classify different objects based solely on their spatial characteristics is crucial for reliable tracking.
To minimize complex- ity and also build from baseline, vehicle detection will be through the PointR- CNN model. The next challenge is to track multiple objects at the same time in real time. This includes linking detected objects to their respective orbits over time, ensuring that the system can track each object’s movement even when they are obscured or temporarily lost from sight. Tracking systems must be able to handle dynamic environments where moving objects are extremely complex, unpredictable, and interact with each other.
In today’s climate, this is especially difficult in real-world driving situations where traffic flows are complex, such as multi-vehicle intersections or turns, and ac- cidents can occur. sudden change in the motion of an object. Systems must not only detect and track objects but also predict their future behavior to support decision-making in autonomous systems.4 Thesis structure There are 5 chapters in this capstone project: • Chapter 1 introduces the importance of 3D multi-object tracking using LiDAR point clouds for automated driving and transportation systems in today’s digital age, and briefly introduces how it works of LiDAR and high- lights its potential to improve safety, efficiency and real-time decision mak- 4 ing. The research aims to develop advanced algorithms for processing, de- tecting and tracking objects in dynamic environments, focusing exclusively on LiDAR data.
• Chapter 2 covers key techniques and metrics used in LiDAR-based 3D object tracking, including deep learning training pipelines, optimization algorithms, and Convolutional Neural Networks (CNN) for data process- ing Point cloud data. It also discusses performance metrics such as IoU, MOTA, and MOTP to evaluate tracking accuracy and consistency. In par- ticular, this chapter also mentions two main algorithms in media tracking, which are Particle Filter-Based Tracking and Particle Filter-Based Track- ing, and compares specific aspects of these two algorithms. • Chapter 3 examines various advanced approaches to 3D multi-object track- ing.
However, to be suitable for evaluating results, these methods will have something in common: they are all evaluated using the KITTI data set. • Chapter 4 delves into proposing a new model in the problem of 3D multi- object tracking (MOT). This chapter highlights the tracking process. The performance of the proposed method is evaluated on three categories (cars, cyclists, pedestrians), demonstrating competitive tracking accuracy and im- proved localization accuracy in a number of figures, although there are some minor differences in other figures.
• Chapter 5 highlights the high accuracy, congestion handling, and track- ing capabilities of the proposed model. Additionally, the model has many limitations including missed detection of small or fast-moving objects and inconsistent tracking in complex situations. Future improvements with the hope of increasing metrics will be the focus. 5 Chapter 2 Theoretical Background In Chapter 2, it presents about preliminary knowledge in this project.1 Deep Learning Deep learning, a subcategory of machine learning, has recently emerged as one of the most influential and transformative technologies in the field of artificial intelligence.
It focuses on the usage of a particular kind of artificial neural net- work known as deep neural networks, which are composed of multiple layers capable of learning and extracting hierarchical features from the raw data auto- matically. Deep learning algorithms perform well while solving complex prob- lems by directly learning from a dataset. Consequently, they yield extremely good performance on challenging tasks. Deep learning has advanced a number of fields considerably since its emer- gence, and it is especially impactful in the areas of computer vision and au- tonomous driving.
Methods in this subfield are an important building block for understanding sensor data like LiDAR point clouds; tasks range from object classification and segmentation to multi-object tracking.2 Neural Networks Neural networks are one of the major building blocks of deep learning, taking inspiration from the way the human brain works.