MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT AUTOMATION AND CONTROL ENGINEERING TECHNOLOGY RESEARCH, DESIGN, AND CONSTRUCT LANE TRACKING AND OBSTACLE AVOIDANCE SYSTEM FOR AUTONOMOUS GROUND VEHICLES BASED ONMONOCULAR VISION AND 2D-LIDAR ADVISOR: ASSOC.PROF LE MY HA STUDENT: LE TRUNG LINH SKL 0 0 9 3 4 2 Ho Chi Minh City, August, 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH-QUALITY TRAINING GRADUATION PROJECT RESEARCH, DESIGN, AND CONSTRUCT LANE TRACKING AND OBSTACLE AVOIDANCE SYSTEM FOR AUTONOMOUS GROUND VEHICLES BASED ON MONOCULAR VISION AND 2D-LIDAR LÊ TRUNG LĨNH Student ID: 18151016 Major: AUTOMATION AND CONTROL ENGINEERING Advisor: LÊ MỸ HÀ, Assoc.Prof Ho Chi Minh City, August 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH-QUALITY TRAINING GRADUATION PROJECT RESEARCH, DESIGN, AND CONSTRUCT LANE TRACKING AND OBSTACLE AVOIDANCE SYSTEM FOR AUTONOMOUS GROUND VEHICLES BASED ON MONOCULAR VISION AND 2D-LIDAR LÊ TRUNG LĨNH Student ID: 18151016 Major: AUTOMATION AND CONTROL ENGINEERING Advisor: LÊ MỸ HÀ, Assoc.Prof Ho Chi Minh City, August 2022 THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- Ho Chi Minh City, August 8, 2022 GRADUATION PROJECT ASSIGNMENT Student name: Lê Trung Lĩnh Student ID: 18151016 Major: Automation and Control Engineering Class: 18151CLA Advisor: Assoc. Prof Lê Mỹ Hà Phone number: 0938811201 Date of assignment: Date of submission: 1. Project title: Research, Design, And Construct Lane Tracking And Obstacle Avoidance System For Autonomous Ground Vehicles Based On Monocular Vision And 2d-Lidar 2. Initial materials provided by the advisor: - Image processing and machine learning documents such as papers and books: - The related thesis of previous students - The hardware specifications and its review.
Content of the project: - Refer to documents, survey, read and summarize to determine the project directions - Collect and visualize data of sensors - Choose models and algorithms for the car’s perception - Write programs for microcontrollers and processors - Test and evaluate the completing system - Write a report - Prepare slides for presenting 4. Final product: the model robot that can operate on the HCMUTE campus has two modes: Autonomic and Manual. CHAIR OF THE PROGRAM ADVISOR (Sign with full name) (Sign with full name) i THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- Ho Chi Minh City, August 8, 2022 ADVISOR'S EVALUATION SHEET Student name: Lê Trung Lĩnh Student ID: 18151016 Major: Automation and Control Engineering Project title: Research, Design, And Construct Lane Tracking And Obstacle Avoidance System For Autonomous Ground Vehicles Based On Monocular Vision And 2d-Lidar Advisor: Assoc. Lê Mỹ Hà EVALUATION 1.
Content of the project: - Design and construct the autonomous model robot that can operate on the HCMUTE campus. - The robot operates based on different techniques and sensors. - The final product fulfills the objectives outlined in this proposition. Strengths: - The automobile model serves as a testing ground for actual automotive improvements to come.
- A control system and algorithms for deep learning were used in the creation of this project. - The program's processing speed is suited for real-time applications. - All of the equipment used in this project are inexpensive. Weaknesses: - The system is unable to achieve full autonomy.
- When operating in an outside setting, the system's accuracy and stability are acceptable. Approval for oral defense? (Approved or denied) .) Ho Chi Minh City, August 8, 2022 ADVISOR (Sign with full name) ii THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- Ho Chi Minh City, August 8, 2022 PRE-DEFENSE EVALUATION SHEET Student name:. Name of Reviewer:. Content and workload of the project.
Approval for oral defense? (Approved or denied) .) Ho Chi Minh City, August 8, 2022 REVIEWER (Sign with full name) iii THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER Student name:. Name of Defense Committee Member:. Content and workload of the project .) Ho Chi Minh City, August 8, 2022 COMMITTEE MEMBER (Sign with full name) iv ACKNOWLEDGEMENTS We want to express our utmost thanks to Professor Le My Ha for his thorough instructions, which provided us with the necessary information to complete this thesis. Despite the period this project requires, it is expected that mistakes will still exist despite our total effort.
With help from our Advisor, especially his input and advice, we hope to gain more experience and succeed in this project topic. We would also like to thank the Faculty of Hight Quality Training and the Faculty of Electrical and Electronics Engineering, where we obtained basic knowledge and experience. Moreover, we would like to thank the members of ISLab members for helping us gain the entire perspective of this project. They shared valuable knowledge and experience with us.
We want to express our gratitude to our families for their support of our team throughout the working of this thesis. Sincere thanks for everything! v A GUARANTEE This thesis is the result of our study and implementation, which we, as a result, formally proclaim. We did not plagiarize from a published article without author acceptance. We will take full responsibility for any violations that may have occurred.
Authors Lê Trung Lĩnh vi ABSTRACT This thesis presented a novel yet simple method for a car overtaking based on a combination between the camera and 2D LiDAR. As for the camera, we utilized two models: "Lane-Line Detection" and "Object Detection." The Lane-Line Detection model plays the plaining path role, which helps the car model determine the next destination in the image frame series. In object detection, YOLOV4 tiny is taken advantage of detecting cars and different types of traffic signs. Moreover, Mosaic augmentation was applied to enhance the performance of the YOLO model.
To boost the inference time and implement a deep learning model on a low-cost device such as Jetson TX2, we converted the two models into TensorRT fp16 format. From the above ideas, the car can be aware of the lane, obstacles, and traffic signs, which will help the vehicle solve many problems on the road. Besides, 2D LiDAR was utilized to check the right side when the camera range was out. Adaptive Breakpoint Detection was applied to cluster the objects in a scanning plane.
Then we find the rule of data by RANSAC and calculate its distance. The estimated distance was the safety condition that helped the car cannot collide with the obstacle. The whole pipeline was conducted with the multithreading technique, which can manage our system and lightly boost the inference time. vii TABLE OF CONTENTS GRADUATION PROJECT ASSIGNMENT.
i PRE-DEFENSE EVALUATION SHEET. iii EVALUATION SHEET OF. iv DEFENSE COMMITTEE MEMBER. vii TABLE OF CONTENTS.
viii LIST OF FIGURES. xi LIST OF TABLES .1 Self-driving Car Technologies. Deep Learning in Autonomous Driving .1 You Only Look Once – YOLO. Dimension cluster for anchor box.
Convolutional Neural Networks (CNNs). Fully Connected layer (FC). Various techniques for evaluating a deep learning model. Random sample consensus algorithm (RANSAC).
TENSOR-RT PATTERN .2 16 Channel PWM controller circuit PCA9685 .3 Devo7 and RX 701.5 IMX335 5MP USB Camera (A). Brushed Motor RC-540PH. RC Servo MG996R. Object Detection Algorithm.
The overview of YOLOv4. YOLOv4-tiny network. Road Lane Detection Algorithm. Design of the Steering Controller .4 Algorithm on 2D LiDAR.
EXPERIMENTS AND RESULTS .1 dataset for lane-line detection .2 dataset for object detection. Lane-line detection.4 COMPARISONS AND EVALUATION. CONCLUSION AND FUTURE WORK. 62 x LIST OF FIGURES Figure 2.
The camera system on MIT self-driving car [4]. LIDAR emits lasers to the environment, receives bounced pulses and calculates distance, makes a point-clouds map, and reconstructs the surrounding environment 5 Figure 2. a) 3D LIDAR sensor; b) Low-cost 2D Lidar. Three main types of object recognition algorithms.
In the left image, some parameters of bounding box detection, including a box of center, height ℎ, and width , the series of point in the right image. The equation of the Intersection over Union. Example of validating the performance of output model. visualize step of the Non-max suppression methods.
YOLOv4 and other cutting-edge object detectors are put to the test [12]. The anchor box in cell i. It has three anchor boxes and one truth bounding box in this cell. The highest overlap with the ground truth bounding box is chosen.
The bounding box of two objects is located on one grid cell. clustering box dimensions on VOC and COCO [6]. Bounding boxes with dimension priors and location prediction. The architecture of Convolutional Neural Networks [31].
The CONV for image (6x6) with 3x3 kernel and stride =1. The result is a 4x4 feature map. Max pooling for 4x4 matrix with 2x2 filter and 2x2 stride. Average pooling for 4x4 matrix with 2x2 filter and 2x2 stride.
Visualization of fully connected layer [13]. Apply zero Padding for input matrix. Convolution for matrix 5x5 with stride = 1. Convolution for matrix 5x5 with stride = 2.
The graph of ReLU [13]. The graph of Leaky ReLU [13]. The confusion matrix in detail. The map of PID operation.
Sample data with no fitting algorithm. The sample result of the least squares technique in the absence of noise. Sample outcome of the least squares method with a few noisy data points. The graphic on the left describes the components that make up a raw data collection.
The illustration on the right provides a RANSAC description of a fitted line. TensorRT uses a variety of optimization techniques to help models run faster. The dynamic range of different precisions. GoogLeNet's Inception module graph is reduced in compute and memory cost because of TensorRT's vertical and horizontal layers integration.
An adaptive Breakpoint Detection algorithm. The result of the ABD algorithm is applied to an autonomous robot system. (a) The system in a real environment with three objects and (b) the output of ABD technique with green representing object 1, yellow representing object 2, purple representing object 3, and pink representing for wall. The overall hardware platform.
16 Channel PWM controller circuit PCA9685. The operation of LED control by PCA9685. RP-Lidar A1 system composition. The RPLIDAR A1 working schematic.
IMX335 5MP USB Camera (A). Pinout of Arduino Uno R3 .4V 5500mAh 70C 2S Lipo Battery. Brushed Motor RC-540PH. RC Servo MG996R.
2D drawing of Servo MG996R. Waveshare touch Screen. block diagram of autonomous vehicle platform. The pipeline of the system.
The structure of CSPDarknet53. Spatial Pyramid Pooling with 3 scales [24]. 6 Illustrations of (a) PAN and (b) modified PAN. The structure of YOLOv4-tiny.
The overall of Lane- line detection algorithm [26]. The estimated straight line distance from LiDAR. The testing road on HCMUTE campus. Custom traffic signs.
The scale model car. Some views of the road on the data. The custom data with different light conditions. The interface of labeling tool.
Training mIoU (left) and loss graph (right)of the lane-line detection model. The loss and mAP graph during training. The result of Lane-Line detection model on test image. The result of the object detection model on test image.
In the left image, the autonomous vehicle system on real environment.The result of the cluster algorithm and RANSAC on the right image. The comparison between 2D RANSAC (left) and Linear Regression (right) on 2D LiDar data. 59 xiii LIST OF TABLES Table 3. Specifications of the car.
Specifications of 16 Channel PWM controller circuit PCA9685. Specifications of Devo7 transmitter. Specifications of RX701. Specifications of RPLidar A1.
Specifications of IMX335 5MP USB Camera (A). Specifications of Jetson TX2 [18]. Specifications of Arduino Uno R3 .