MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT AUTOMATION AND CONTROL ENGINEERING DESIGN AND IMPLEMENTATION OF A LANEKEEPING AN NAVIGATION SYSTEM FOR SELFDRIVING VEHICLE BASED ON THE FUSION OF CAMERA, GPS AND IMU LECTURER: LÊ MỸ HÀ, Assoc. STUDENT: NGUYỄN HOÀNG HẢI NAM PHẠM DUY HƯNG SKL009344 Ho Chi Minh City, August, 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT DESIGN AND IMPLEMENTATION OF A LANE KEEPING AND NAVIGATION SYSTEM FOR SELF- DRIVING VEHICLE BASED ON THE FUSION OF CAMERA, GPS AND IMU NGUYỄN HOÀNG HẢI NAM - 18151021 PHẠM DUY HƯNG - 18151188 Major: AUTOMATION AND CONTROL ENGINEERING Advisor: LÊ MỸ HÀ, Assoc. Ho Chi Minh City, August 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT DESIGN AND IMPLEMENTATION OF A LANE KEEPING AND NAVIGATION SYSTEM FOR SELF- DRIVING VEHICLE BASED ON THE FUSION OF CAMERA, GPS AND IMU NGUYỄN HOÀNG HẢI NAM - 18151021 PHẠM DUY HƯNG - 18151188 Major: AUTOMATION AND CONTROL ENGINEERING Advisor: Assoc. LÊ MỸ HÀ Ho Chi Minh City, August 2022 THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- Ho Chi Minh City, August 5, 2022 GRADUATION PROJECT ASSIGNMENT Student name: Nguyễn Hoàng Hải Nam Student ID: 18151021 Student name: Phạm Duy Hưng Student ID: 18151188 Major: Automation and Control Engineering Class: 18151CLA2 Advisor: Assoc.
Lê Mỹ Hà Phone number: 0938811201 Date of assignment: Date of submission: 1. Project title: Design and implementation of a lane keeping and navigation system for self- driving vehicle based on the fusion of camera, GPS and IMU. Initial materials provided by the advisor: - Documents and articles related to machine learning and image processing. - The related thesis of previous students.
- The hardware specifications and its review. Content of the project: - Read, perform surveys, summarize to determine the scope of the project. - Research the theoretical array math related to self-driving cars to give directions and handle problems during model building. - Read and process sensors signal.
- Choose model and algorithm for lane keeping of vehicle. - Write program to control microcontroller. - Research fusion algorithm for sensors. - Write project report.
- Prepare slide for presenting. Final product: The car model can operate inside HCMUTE school campus based on a combination of camera, GPS and IMU in Auto and Manual mode under not too complex conditions. CHAIR OF THE PROGRAM ADVISOR (Sign with full name) (Sign with full name) i THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- Ho Chi Minh City, August 5, 2022 ADVISOR’S EVALUATION SHEET Student name: Nguyễn Hoàng Hải Nam Student ID: 18151021 Student name: Phạm Duy Hưng Student ID: 18151188 Major: Automation and Control Engineering Project title: Design And Implementation Of A Lane Keeping And Navigation System For Self- Driving Vehicle Based On The Fusion Of Camera, GPS And IMU. Lê Mỹ Hà EVALUATION 1.
Content of the project: - The content of this report is 64 pages. - The design and construction of car model that can operate HCMUTE school campus in Auto and Manual mode. - The system runs based on a series of different sensors and algorithms. - The final product meets the requirements in the proposal.
Strengths: - The car model creates a pipeline for future implementation on life size car. - Project was designed with machine learning-based image processing imbued with controller technique. - All the sensors used for this project are low-cost. - Process time of program is suitable for real-time applications.
Weaknesses: - The system cannot reach full automation level instead requires operator presence due to there are some improvements to be made. - The accuracy and stability of system stops at acceptable level in outdoor environment due to all sensors are low-cost. Approval for oral defense? (Approved or denied) .) Ho Chi Minh City, August 5, 2022 ADVISOR (Sign with full name) ii THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- Ho Chi Minh City, August 5, 2022 PRE-DEFENSE EVALUATION SHEET Student name:. Name of Reviewer:.
Content and workload of the project. Approval for oral defense? (Approved or denied) .) Ho Chi Minh City, month day, year REVIEWER (Sign with full name) iii THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -------- EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER Student name:. Name of Defense Committee Member:. Content and workload of the project .) Ho Chi Minh City, month day, year COMMITTEE MEMBER (Sign with full name) iv ACKNOWLEDGEMENTS We would like to express our utmost thanks to Professor Le My Ha for his thorough instructions which provided us with necessary information to complete this thesis.
It is expected despite the period this project requires and our full effort mistakes will still exist. With help from our advisor’s especially his inputs and advice we hope to gain more experience and achieve success in this project topic. We would also like to extend to thank Faculty of Hight Quality Training and Faculty of Electrical and Electronics Engineering where we obtained basic knowledge and experience. Moreover, we would like to thank the members of ISLab member for helping us gaining the full perspective of this project.
They shared valuable knowledge and experience with us. We would like to express our gratitude to our families for their support of our team throughout the working of this thesis. Sincere thanks for everything! v A GUARANTEE This thesis is the result of our study and implementation, which we hereby formally proclaim. We did not plagiarize from a published article without author acceptance.
We will take full responsibility for any violations that may have occurred. Authors Nguyễn Hoàng Hải Nam Phạm Duy Hưng vi ABSTRACT This thesis proposes a self-driving system for Autonomous Ground Vehicle in urban area based on the fusion between Camera-based Image Segmentation method and a GPS-and-IMU-based lateral controller. For the task of Image Segmentation, we use a Segmentation Model called BiseNet to segment images coming from the camera and derive a heading position for the vehicle using the Edge-detected segmented images and geometric operations. To navigate the vehicle using GPS signals, we design a waypoint- based lateral controller to steer the vehicle with respect to a designated GPS track.
Our GPS measurements are collected, corrected, and updated with the help of the Extended Kalman Filter. The fusion between the two controllers is flag-based and used to derive the final steering control of the vehicle in different road scenarios. The system includes a NVIDIA Jetson TX2 development board stacked on the rear top of a 1/10 scale RC car along with a 130 POV camera, a smart phone to stream sensor signals through Wi-Fi. All experiments were carried out on a pre-defined road track along HCMCUTE campus where the system was able to run autonomously in real-time at 11-13 fps.
vii TABLE OF CONTENTS ACKNOWLEDGEMENTS. vii TABLE OF CONTENTS.viii LIST OF FIGURES. xi LIST OF TABLES. xvi CHAPTER 1: INTRODUCTION.
2 CHAPTER 2: THEORETICAL BASIS. Technologies used in Self-driving Car. Deep Learning Applications in Self-driving Car. Convolutional Neural Network.
Stride and Padding. Fully Connected Layer. Image Segmentation Techniques. Threshold Based Segmentation.
Edge Based Segmentation. Clustering Based Segmentation. Artificial Neural Network Based Segmentation. Semantic Segmentation Dataset for Road Scene Application.
Our Customized Dataset. 27 CHAPTER 3: DESIGN AND CALCULATION. Design and Propose of the Image Segmentation Model. Mathematical Analysis of a 4-wheeled Ground Vehicle.
Ackerman Steering Geometry. Analyze the Kinematic Model of Four-wheeled Vehicle. Analyze the Dynamic Model of Four-wheeled Vehicle. Designing of Control System.
Waypoint-based Geometric Lateral Controller. Image-Segmentation-based Lateral Controller. Preparing of GPS Road Map. Extended Kalman Filter.
Fusion Strategy between Segmentation-based Controller and GPS-waypoint Controller. 49 CHAPTER 4: HARDWARE ASSEMBLY. Brushed Motor RC-540. MG996R Servo Motor.
PCA9685 Servo Driver. Jetson NVIDIA TX2. Self-driving Car Model. 58 CHAPTER 5: EXPERIMENTAL RESULTS.1 Image Segmentation Network result.2 Real-time Navigation Results of Self-driving Car.
62 CHAPTER 6: CONCLUSION AND FUTURE WORK.2 Work in the future.66 x LIST OF FIGURES Figure 2. Camera on self-driving car [5]. GPS tracking for Ground vehicles [7]. a) IMU sensor based on three types of sensors.
b) IMU sensor on plane. c) IMU sensor in mobile phone. End-to-end learning method for self-driving car. Input image is processed through a stack of Convolution Layers that learns useful features which is then flattened to join a Fully Connected network and output control signal.
Image segmentation for road scene understanding. The above figure demonstrates the segmentation process using an Encoder-Decoder based Convolutional Network. Object Detection in Self-driving applications. Example of a One-stage detector – The YOLO architecture [8].
The DarkNet architecture act as a feature extractor. CNN architecture for handwriting classification [10]. Convolution of input image size 28x28 and kernel size 3x3 [11]. Example of ReLU [11] activation function applied on image size 26x26.
Max Pooling 2x2 on input image 26x26 creates output 13x13 [11]. a) original image; b) Background image; c) Image after background subtraction; d) Threshold image [12]. Real-time adaptive image thresholding [13]. Different type of edges.
a) Step edge; b) Roof edge; c) Ramp edge; d) Line edge. Type of edge detector. a) Sobel; b) Prewitt; c) Roberts. An example of edge-based segmentation.
a) Original image; b) Edge detection [14]. Segmentation using K-Mean Clustering applied on an RGB image with different k value. a) Origin image; b) Segmented image with k=2; c) Segmented image with k=3. Example of K-Mean clustering method applied on 2D data.
Image segmentation network according to the encoder/decoder structure [15]. The arrows display the operations of network, red is downsampling, green is upsampling and blue is skip connections. Downsampling in convolution neural network. CNNs network with an upsampling layer [16].
Residual Block structure [17]. Inception network structure with division matrices of 1x1 and matrix to find correlation points of 3x3 [18]. Overview of the Extreme topology of the Inception network with a 3x3 spatial convolution layer for each 1x1 convolutional output [18]. Detailed structure of the Xception network.
Data goes into Entry flow, comes out Middle flow repeats 8 times and comes out Exit Flow. All Convolution and Separable Convolution operations are batch-normalization. The Separable Convolution operation uses a 1x1 division matrix [18]. Model Size vs ImageNet accuracy.
All numbers are for single-crop, single- model [19]. The FCN [20] model appears to be a repurposing of the famous AlexNet model which was trained on ImageNet dataset. An issue in FCN paper, the coarse segmentation appears to be poorly detailed. The author address this by adding Skip-connections.
a) Ground truth image; b) Predicted segmentation. The FCN architecture. Encoder module consists of an AlexNet, and decoder module introduce Skip connections. The segmentation is much better after the integration of Skip connections.
a) Ground truth image; b) Predicted segmentation. UNet network structure with kernel down-sampling is 3x3 and up-sampling is 2x2 [21]. The input image size is 572x572 and the output image size is 388x388. UNet's segmentation result on ISBI cell tracking challenge [21].
The complete architecture of the ENet model is shown. The model includes both encoder (light blue) and decoder (dark blue) parts. The upward and downward arrows indicate up-sampling and down-sampling operations. Right hand arrows show different types of convolutions including normal, dilated, and asymmetric.
Up and down arrows show up-sampling and down-sampling operations respectively [22]. The CamVid dataset contains more than 30 labels that correlate to 30 items such as automobiles, roads, sidewalks, light poles, people, and so on [23]. Cityscapes dataset with 30 semantics label classes [24]. Custom dataset comprising of 1299 images collected from a vehicle- mounted camera on the HCMUTE campus, labels are tagged using Hasty.
a) Afternoon data block C; b) Morning data block C; c) Night data block A; d) Afternoon data block A. Data augmentation methods used in training steps random resize, random rotate, random perspective transform, and color jitter. Left: original heavily distorted image. Right: calibrated image.