MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT AUTOMOTIVE ENGINEERING VEHICLE DETECTION, TRACKING AND BEHAVIOR ANALYSIS WITH ENHANCING DEPTH INFORMATION ADVISOR: MSc. NGUYEN TRUNG HIEU STUDENT: HA PHAN NGOC QUAN TRUONG THANH NGUYEN SKL010782 Ho Chi Minh city, July 2023 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT VEHICLE DETECTION, TRACKING AND BEHAVIOR ANALYSIS WITH ENHANCING DEPTH INFORMATION HA PHAN NGOC QUAN Student ID: 19145008 TRUONG THANH NGUYEN Student ID: 19145158 Major: AUTOMOTIVE ENGINEERING Advisor: NGUYEN TRUNG HIEU, MSc. Ho Chi Minh City, July 2023 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT VEHICLE DETECTION, TRACKING AND BEHAVIOR ANALYSIS WITH ENHANCING DEPTH INFORMATION HA PHAN NGOC QUAN Student ID: 19145008 TRUONG THANH NGUYEN Student ID: 19145158 Major: AUTOMOTIVE ENGINEERING Advisor: NGUYEN TRUNG HIEU, MSc. Ho Chi Minh City, July 2023 THE SOCIALIST REPUBLIC OF VIETNAM Independence :.
Freedom- Happiness Ho Chi Minh City, July OJ, 2023 GRADUATION PROJECT ASSIGNMENT Student name: Ha Phan Ngoc Quan Student ID: 19145008 Student name: Truong Thanh Nguyen Student ID: 19145158 Major: Automotive Engineering Class: 19145CLA Advisor: Nguyen Trung Hieu, MSc. Phone number: 096 2497 102 Date of assignment: March 3, 2023 Date of submission: July 1, 2023 1. Project title: Vehicle Detection, Tracking and Behavior Analysis with Enhancing Depth. Initial materials provided by the advisor: _______ ___ ___ 3.
Content of the project: Desel -:p IL JJgd. Final product: -�Oud:1._, -------- CHAIR OF THE PROGRAM ADVISOR (Sign withfull name) (Sign withfull name) THE SOCIALIST REPUBLIC OF VIETNAM Independence - Freedom- Happiness Ho Chi Minh City, July OJ, 2023 ADVISOR'S EVALUATION SHEET Student name: Ha Phan Ngoc Quan Student ID: 19145008 Student name: Truong Thanh Nguyen Student ID: 19145158 Major: Automotive Engineering Project title: Vehicle Detection, Tracking and Behavior Analysis with Enhancing Depth Information Advisor: Nguyen Trung Hieu, MSc. Content of the project: �\�. Approval for oral defense? (Approved or denied) .) Ho Chi Minh City, month day, year ADVISOR (Sign with full name) ./_\,,/ THE SOCIALIST REPUBLIC OF VIETNAM Independence - Freedom- Happiness Ho Chi Minh City, July 01, 2023 PRE-DEFENSE EVALUATION SHEET Student name: Ha Phan Ngoc Quan Student ID: 19145008 Student name: Truong Thanh Nguyen Student ID: 19145158 Major: Automotive Engineering Project title: Vehicle Detection, Tracking and Behavior Analysis with Enhancing Depth Information Name of Reviewer:.
Content and workload of the project .fui✓• js,0:;i ••••fl. Approval for oral defense? (Approved or denied) .) Ho Chi Minh City, month day, year Jil,_JlJ1 zoL!:, REVIEWER d (Sign withfull name) Acknowledgment We would like to express our sincere gratitude to our advisors, Mr. Nguyen Trung Hieu, MSc. Tran Vu Hoang, PhD.
for their invaluable guidance and support throughout the research process. We also wish to thank the Vehicle Automatic Control Laboratory, HCMUTE for facility support, as well as the Lab’s collaborators for providing us the best conditions to accomplish our project. i Table of Contents Abstract. viii Chapter 1: INTRODUCTION.
Research and Literature Review. Data Collection and Preprocessing. Vehicle Detection and Driving Scene Recognition. Behavior Analysis and Warning.
Warning and Alert System. Documentation and Reporting. Iterative Development and Enhancement. False Positives and False Negatives.
Generalization to Unseen Scenarios. Human Factors and Driver Interaction. 7 Chapter 2: LITERATURE REVIEW. Data Fusion Mechanism.
Stereo Disparity Block Matching. Distance Estimation based on Depth Information. Multi-tasking Detection Model. Motion-based Tracking.
Feature-based Tracking. Vehicle Behavior Analysis/Trajectory Prediction. Behavior Analysis Problem Statement. Evaluation Metrics and Losses.
29 Chapter 3: MULTI-TASKING MODEL FOR PANOPTIC DRIVING PERCEPTION. Network Model Selection. 35 Chapter 4: VEHICLE TRACKING. Feature-based Tracker.
Data Association and Track Management. 41 Chapter 5: VEHICLE BEHAVIOR ANALYSIS AND DRIVER WARNING. Post Processing of Multitasking Detection Model output. Line merging and filtering.
Establish Warning Region of Interest. Predict Future State using Kalman Filter. Analyzing and Warning. Case 1: Warning of Decelerating.
Warning of Lane-changing. Warning at Intersection. Behavior Analysis and Warning. 50 Chapter 7: CONCLUSION AND FUTURE WORK.
53 iv List of Figures Fig. 1 Pixel level disparity between left and right image. 2 Stereo disparity matching output of SGBM. 3 Distance estimation model.
4 The data sources are geographically distributed across various cities and regions in highly populated areas of the United States. Each dot on the map represents the starting location of a video clip in BDD100K dataset. 5 Overview of BDD100K dataset for multitasking purpose. 6 Instance statistics of our object categories.
(a) Number of instances of each category, which follows a long-tail distribution. (b)Roughly half of the instances are occluded. (c) About 7% of the instances are truncated. 7 The confusion matrix.
8 Dice coefficient and IoU (Jaccard index) can be calculated based on the overlapping area between the two circles and the total area covered by the circles. 9 Schematic Description of the Kalman Filter Algorithm [20]. 10 Mounting position of sensors with respect to the vehicle body [2]. 11 In reference to [30], the modeling of trajectory predictions for dynamic agents on the road involves considering their interaction-awareness and road awareness.
The trajectory of one vehicle, referred to as Veh.2, is dependent on the trajectory of another vehicle, denoted as Veh.3, and vice versa. 12 Lane Coordinate System. The diagram explains the setup of three coordinate systems: global map coordinate system, ego-vehicle body-fixed coordinate system, and lane coordinate system. 13 Illustration of future vehicle localization.
Location and scale are represented as bounding boxes in predictions. 14 At each time step t, the precision and coherence of the predicted bounding boxes for all traffic participants from preceding frames are assessed. Based on this evaluation, anomaly score for the scene is given. 15 Overview of the complete framework: vehicle detection and scene recognition, vehicle tracking and behavior analysis with enhancing depth information.
16 Day time perception result comparison. 17 Night time perceptive result comparison. 18 The network architecture of YOLOPv2. 19 Methodology flow chart.
20 System Overview of Tracking Framework. 21 Initialize component of tracker. 22 Track component of Tracker. 23 Overview of the Behavior Analysis and Warning component.
24 Post-processing pipeline for the output of the Multitasking Detection Model. 25 Applying Kalman Filter prediction to determine the next positions of tracked vehicles. Estimated positions are shown in white bounding boxes, while the current state in yellow ones. 26 The vehicle with decelerating action in the ego-lane is marked as warning.
27 The vehicle performing lane-changing to ego-lane is marked as warning. 28 At intersections, all the vehicles in any drivable area are marked as warning. 29 A tracked vehicle is marked as warning due to its deceleration based on distance data. 30 Additional results of behavior analysis and warning.
31 A vehicle whose predicted position intersects ego-lane area is being warned. 32 At intersection, any vehicle inside drivable area is warned. 51 vi List of Tables TABLE I. Performance Comparison among Panoptic Driving Perception Networks [14].
Evaluation Results Of Three Methods On MOTA Metric. Evaluation Results Of Three Methods On IDF1 Metric. Evaluation Results Of Three Methods On HOTA Metric. 49 vii Abstract Vehicle behavior analysis and warning systems play a crucial role in Advanced Driver Assistance Systems (ADAS) applications to enhance road safety and driver awareness.
This project presents a comprehensive overview of vehicle behavior analysis techniques and their application in ADAS. It explores various sensors and data sources used to collect information about the vehicle and its surrounding environment. Additionally, the project discusses different algorithms and models employed to detect, track and analyze vehicle behavior, as well as traffic scenes; including lane departure detection, collision prediction, and abnormal maneuver identification. Furthermore, the integration of these techniques into a warning system is examined, focusing on the generation of timely and effective alerts to the driver.
The paper concludes with a discussion on the challenges and future directions in vehicle behavior analysis and warning systems for ADAS, highlighting the potential for advancements in sensor technology, deep learning, and human-machine interaction to further improve road safety and driver assistance. Keywords: ADAS, Advanced Driver Assistance Systems, data-fusion, panoptic driving perception, vehicle tracking, vehicle behavior analysis viii Chapter 1: INTRODUCTION 1. Topic Reasoning As automotive technology continues to advance, the integration of Advanced Driver Assistance Systems (ADAS) has become increasingly vital in the pursuit of safer and more efficient transportation. ADAS applications are revolutionizing the way vehicles interact with their surroundings, providing drivers with enhanced safety features and intelligent assistance.
These systems are designed to mitigate risks, reduce accidents, and enhance the overall driving experience. In this context, the need for ADAS arises from the pressing demand to improve road safety, optimize vehicle behavior, and pave the way for a future of autonomous driving. The primary driving force behind the need for ADAS applications is the paramount goal of enhancing safety on the roads. With millions of vehicles navigating complex traffic scenarios daily, the risk of accidents and collisions remains a significant concern.
For instance, as reported in [1], human error during lane change, speeding and tailgating caused 41.0% of total severe crashes. Ultimately, ADAS technologies step in to address this challenge by providing real-time monitoring, analysis, and intervention capabilities. These systems employ a variety of sensors, cameras, and algorithms to detect potential hazards, warn the driver, and even autonomously intervene when necessary. By augmenting human perception and reaction times, ADAS applications aim to prevent accidents, minimize their severity, and ultimately save lives.
Furthermore, the need for vehicle behavior analysis reinforces the importance of ADAS applications. Understanding and optimizing the behavior of vehicles on the road is crucial for ensuring both safety and efficiency. Vehicle behavior analysts play a vital role in studying and evaluating the dynamic interactions between vehicles, road conditions, and drivers. By leveraging ADAS technologies, these analysts can gather real-world data, analyze driving patterns, and identify areas for improvement.
This analysis can inform the development and refinement of ADAS algorithms, leading to more accurate detection of potential risks and more effective interventions. Ultimately, this synergy between ADAS applications and vehicle behavior analysis helps to create a safer and more intelligent driving environment for all road users. Our framework for vehicle detection, tracking, and behavior analysis is a comprehensive solution that addresses the demands for enhanced road safety and intelligent driver assistance. By leveraging advanced computer vision and machine learning techniques, our framework is capable of accurately detecting and tracking vehicles in real-time, while also analyzing their behavior.
This enables us to provide timely warnings to drivers about abnormal behavior exhibited by other cars on the road. 1 By integrating vehicle detection, tracking, and behavior analysis into a unified framework, we provide drivers with a comprehensive tool to enhance their situational awareness and safety on the road. Our system's ability to identify and notify drivers about abnormal behaviors exhibited by other cars can significantly reduce the risk of accidents and improve overall road safety. Objectives The aim of the project is to develop a robust and intelligent framework for vehicle detection, tracking, and behavior analysis that addresses the need for enhanced road safety and driver assistance.
The primary goal is to combine the data availability from various sensors and leverage advanced computer vision and machine learning techniques to accurately detect and track vehicles in real-time, while also analyzing their behavior in recognized traffic scene to identify abnormal patterns. By achieving this aim, we aim to provide timely warnings and alerts to drivers about potential risks and hazards posed by other vehicles on the road. Furthermore, the project aims to enhance overall road safety by utilizing state-of-the-art algorithms and methodologies for panoptic driving perception.