MINISTRY OF EDUCATION AND TRAINING DUY TAN UNIVERSITY ADAPTIVE LEARNING SOLUTION BASED ON DEEP LEARNING FOR TRAFFIC OBJECT RECOGNITION DOCTOR OF PHILOSOPHY OF COMPUTER SCIENCE Da Nang, 2022 luan an MINISTRY OF EDUCATION AND TRAINING DUY TAN UNIVERSITY ADAPTIVE LEARNING SOLUTION BASED ON DEEP LEARNING FOR TRAFFIC OBJECT RECOGNITION Major: Computer Science Code: 9480101 Da Nang, 2022 luan an i COMMITMENT To the best of my knowledge, I hereby certify that all the content in the thesis entitled "Adaptive learning solution based on deep learning for traffic object recognition" is my own research. The figures and results of the thesis are honest, fully quoted and have not been previously published by another. The author's signature luan an ii ACKNOWLEDGEMENTS First of all, I would like to express my endless thanks to my instructors. Their kindly support and advices went through the completion process of my PhD thesis.
Their companion encouraged me to improve my work. Their instructions and motivation helped me to grow as a research scientist. I would also like to thank my council reviewers, members and independent scientists for giving me contribution and brilliant comments to my thesis. I would like to express my sincere thanks to the Board of Trustees and Board of Rector of Duy Tan University, the teachers and officers of Duy Tan University's Graduate School, for helping me in the process of learning and researching at University.
I also acknowledge my thankfulness to the Board of Directors of the Quang Binh provincial Department of Information and Communications for kind assistances and support in my work and learning so that I can achieve the results today. Many thanks come to the research group’s members for their participation in the published works and allowing me to use the research results for this thesis. Finally, my deeply thanks come to my loved people and friends who were always beside me to help me when I need for the last time. A special thanks to my family where I got the most assistances and motivation for the whole of my life.
In spite of the fact that many efforts are made during the working process, the thesis may remain shortcomings due to limited time and research conditions. All valuable comments and suggestions for the thesis completion will be highly appreciated. The author luan an iii TABLE OF CONTENTS LIST OF FIGURES. vi LIST OF TABLES .viii LIST OF ABBREVIATIONS.
Research subject and scope. The structure of the thesis. OVERVIEW OF ARTIFICIAL INTELLIGENCE .1 Overview of artificial intelligence. Definition of artificial intelligence .2 History of artificial intelligence.
Machine learning and identification techniques .1 Machine learning applications. Video games and robotics .2 Basic recognition techniques in machine learning .4 Support vector machine.5 Artificial neural network .3 Deep Learning and Adaptive Learning .1 Overview of Deep Learning and Adaptive Learning .2 Deep neural network (DNN) .3 Convolution neural network (CNN). 17 luan an iv 1.4 Domestic and international research. RECOGNIZING OBJECTS BY DEEP LEARNING .1 Object recognition problems .1 Problem: Pedestrian action prediction .2 Problem: Vehicle recognition .1 Solution to pedestrian recognition .1 Extracting features and training classifier model .2 Pedestrian action prediction .2 Solution to vehicle recognition .1 Sequential Deep Learning architecture .1 Extracting features and training classifier model .2 Pedestrian detection and action prediction .3 Categorical vehicle recognition.
43 CHAPTER 3: DEVELOPMENT OF ADAPTIVE LEARNING TECHNIQUE IN OBJECT RECOGNITION .1 Adaptive learning problem in object recognition.1 Overview of solutions .1 Concept Definitions of System Components .2 General Structure of the System .3 Details of the Proposed Architecture .1 Training CNN Model .2 Retraining and updating model. OPTIMIZING HYPERPARAMETERS IN ADAPTIVE LEARNING .1 Problem of optimizing hyperparameters .1 Training the initial PDNet model .2 Optimization of learning parameters, update PDNet model .3 Compare with the state - of – the - art models. 95 CONCLUSION AND DEVELOPMENT DIRECTION. 98 LIST OF PUBLISHED SCIENTIFIC WORKS RELATED TO THE THESIS.
101 luan an vi LIST OF FIGURES Figure 1.1 History of artificial intelligence .2 Classification simulation of SVM .3 Illustration of neural network architecture .4 Simple Deep Learning network with one layer and Deep Learning network with multiple hidden layers .5 Architecture of a simple convolution neural network .1 The process of extracted features by CNN model from image dataset .2 The process of pedestrian movement prediction .3 Proposed vehicle detection model .4 Input images and simulate rich features of image .5 Influence of other objects on the road on pedestrian movement prediction .6 Example input image for recognition .7 Pedestrian detection with scores = 0.8 ROI extraction from pedestrian image .9 The order of classifications of pedestrians when there are many pedestrians on the road in an input image.10 Some examples of vehicle categories .11 Pedestrians detected and ROI extracted .12 The weight values of the filter of the first convolution layer. This layer consists of 64 filters size 7x7, each of which is connected to three RGB image input channels .13 Some results of linear convolution and linear correction for the input images being motors .14 Comparison of HOG+SVM, CNN model and CNN with augmenting data .1 General flowchart of the system .2 Simulation of training dataset, consisting of (a) original image set and (b) labeled set .3 Simulation of extracting Region of interest .4 PDNet model structure .5 Simulation of tracking process of objects .6 Training progress of PDNet-Vehicle0 model .7 Training progress of PDNet-TrafficSign0 model .8 Comparing the accuracy of recognition results of retrained Vehicle and Traffic sign model .9 Comparison results of our proposed approach and other methods .10 Comparison results by applying our Adaptive Learning to other methods .1 Stimulation of searching way of Hyperparameter values by Grid Search (a) and Random Search (b) (Source: Medium.2 Operation model of Bayesian optimization .3 Gaussian process (Source: https://www.net/profile/Akshara_Rai)72 Figure 4.4 Overall proposed model .5 Operating model of the Bayesian algorithm. 78 luan an vii Figure 4.6 The confusion matrix of the accuracy of initial PDNet-Vehicle and PDNet- TrafficSign model .7 The Bayesian function's objective value evaluated on objective function evaluations .8 The confusion matrix for test data in the search process of optimal hyperparameter and model.9 The confusion matrix of the accuracy of PDNet-Vehicle1 and PDNet- TrafficSign1 model .10 The confusion matrix of the accuracy of PDNet-Vehicle2 and PDNet- TrafficSign2 model .11 Comparing the accuracy of recognition results of Vehicle and Traffic sign model .12 The confusion matrix of the accuracy of AlexNet model for vehicle recognition .13 The confusion matrix of the accuracy of AlexNet model for traffic sign recognition .14 The chart showing the increasing accuracy on recognition of AlexNet model after the updated recognition model with optimal hyperparameters applied .15 The confusion matrix of the accuracy of Vgg model for vehicle recognition .16 The confusion matrix of the accuracy of Vgg model for traffic sign recognition .17 The chart showing the increasing accuracy on recognition of Vgg model after the updated recognition model with optimal hyperparameters applied. 94 luan an viii LIST OF TABLES Table 2.1 CNN architecture with 22 hidden layers, 1 input layer, and the final classification layer .2 Image and label datasets of extracted and trained features .3 Maximum confusion matrix for pedestrian action prediction .5 Training data after augmentation and balance data .6 Confusion matrix of vehicle recognition using HOG and SVM .7 Confusion matrix of vehicle recognition using CNN .8 Confusion matrix of vehicle recognition using CNN and data augmentation .1 The color map .2 The vehicle objects serving recognition by PDNet model .3 The traffic objects serving recognition by PDNet model .4 Images and labels dataset to train PDNet1 .5 Global accuracy of IONet model.
6 Accuracy of objects of IONet model .7 Image datasets for testing PDNet-TrafficSign model .8 Image datasets for testing PDNet-Vehicle model. 9 Image datasets for training PDNet-Vehicle .10 The confusion matrix of the accuracy of PDNet-Vehicle0 model .11 Image datasets for training PDNet-TrafficSign .12 The confusion matrix of the accuracy of PDNet-TrafficSign0 model .13 The configuration of the device to test the process speed .14 Image data for retraining PDNet-Vehicle0 model.15 Image data for retraining PDNet-TrafficSign0 model .16 Image data for retraining PDNet-Vehicle1model.17 Image data for retraining PDNet-TrafficSign1 model .18 The confusion matrix of the accuracy of PDNet-Vehicle1 model .19 The confusion matrix of the accuracy of PDNet-TrafficSign1 model .20 The confusion matrix of the accuracy of PDNet-Vehicle2 model .21 The confusion matrix of the accuracy of PDNet-TrafficSign2 model .22 Comparing the processing speed on traffic sign and vehicle sign between our proposed model and AlexNet,Vgg model.1 PDNet model structure and parameters .2 Hyperparameters in the training process of CNN (Training option) .5 The object for PDNet model recognition.3 Image datasets for testing the PDNet-Vehicle model .4 The object for PDNet model recognition. 6 Image datasets for testing PDNet-TrafficSign model .7 model Parameter domain values .9 Image datasets for training initial PDNet-Vehicle.8 The configuration of the device. 81 luan an ix Table 4.10 Image datasets for training initial PDNet-TrafficSign.11 Image data (Data-Vehicle0) for searching hyperparameters and the PDNet- Vehicle1 model .12 Image data (Data-TrafficSign0) for searching hyperparameters and the PDNet- TrafficSign1 model .13 Found optimal hyperparameter values of PDNet-Vehicle1 and PDNet- TrafficSign1 model .15 Image data (Data-TrafficSign1) for searching hyperparameters and the PDNet- TrafficSign2 model .14 Image data (Data-Vehicle1) for searching hyperparameters and the PDNet- Vehicle2 model .16 Found optimal hyperparameter values of PDNet-Vehicle2 and PDNet- TrafficSign2 model.
17 Results of proposed methods compared to those of the Chapter 3. 95 luan an x LIST OF ABBREVIATIONS Abbreviation Explanation AI Artificial Intelligence HAC Human Action Recognition ML Machine Learning DL Deep Learning AL Adaptive Learning CNN Convolution Neural Network NN Neural Network DNN Deep Neural Network ANN Artificial Neural Network SVM Support Vector Machine RF Random Forest ACF Aggregate Channel Features ITS Intelligent Transportation Systems ROI Region of interest SaaS Software-as-a-Service ADAS Advanced Driver Assistance Systems HOG Histograms of Oriented Gradients AV Auto Vehicle luan an 1 INTRODUCTION 1. Introduction Artificial intelligence (AI) is intelligence demonstrated by an artificial system. Artificial intelligence is everywhere today such as office applications, automatic answering systems, intelligent traffic management, smart home management, etc.
Since the Computer hardware systems became increasingly capable, artificial intelligence has made great progress, applied more widely in all fields of life and society. Artificial intelligence focuses on developing algorithms and applications that support human in decision making or self- decision making in the process of data identifying and acquiring. Object detection, Object action recognition and Human action recognition are one of the research targeted directions such as security surveillance systems, security, manual remote control systems, blind assist systems, sports data analysis systems, automated robots, self-driving cars [1, 2, 3, 4, 5], and so on. There have been many studies proposing many different solutions to artificial intelligence development such as heuristic algorithm, evolution algorithm, Support Vector Machine algorithm, Hidden Markov Model algorithm, expert method, neural network method, [6, 7, 8], etc.
Traditional solutions, yet all require human intervention and huge amounts of data to analyze and store but low accuracy and limited identification cases. To overcome those shortcomings, machine learning with focusing on Deep Learning Method (Deep Learning) is now being applied in artificial intelligence in terms of object detection and action recognition. Deep Learning has been a hotly debated AI topic. As a small category of machine learning, Deep Learning focuses on solving issues related to artificial neural networks in order to upgrade technologies such as voice recognition, image recognition and natural language processing.
In just a few years, Deep Learning has promoted progress in a variety of fields which are used to be very difficult to luan an 2 artificial intelligence researchers such as Object Perception, Machine Translation, voice recognition, etc. However, despite of the fact that issues related to AI were solved, Deep Learning has still remained limitations that need to be settled.