VIETNAM NATIONAL UNIVERSITY, HANOI VNU UNIVERSITY OF SCIENCE ——————– * ——————— VŨ MINH HƯNG DEEP LEARNING APPROACH FOR DRIVER BEHAVIOR ANALYSIS MASTER’S THESIS HANOI, 2023 VIETNAM NATIONAL UNIVERSITY, HANOI VNU UNIVERSITY OF SCIENCE ——————– * ——————— VŨ MINH HƯNG DEEP LEARNING APPROACH FOR DRIVER BEHAVIOR ANALYSIS Council: Data science Code: 8904648.01QTD MASTER’S THESIS SUPERVISOR: PhD. Đỗ Thanh Hà HANOI, 2023 Declaration I, Vũ Minh Hưng, hereby declare that the information presented in this thesis is true and correct to the best of my knowledge. The content of this thesis, including text, illustrations, data, plots, and pictures, is original and is the outcome of my research work. Any relevant material taken from the open literature has been referred to and cited in accordance with established ethical norms and practices.
Acknowledgments I express my deepest gratitude to my supervisor, PhD. Đỗ Thanh Hà, for enlightening me with the essence of science and the art of research. Her invaluable guidance has been instrumental in shaping my research journey, enabling me to embark on a path of excellence. This thesis stands as an evidence to her unwavering support and mentorship.
Additionally, I extend my heartfelt appreciation to PhD. Lê Hồng Phương for granting me access to the essential hardware resources that have played a pivotal role in the successful completion of my thesis. Lastly, I would like to acknowledge my family for their boundless love and unwavering encouragement throughout this endeavor. This dissertation is a reflection of their immeasurable impact on my life.
Related Works for Driver Behavior Analysis 6 2. Classical machine learning approaches. Using non-visual features. Using visual features.
Deep learning approaches. Using non-visual features. Using visual features. Contrastive Learning Framework and its Improvements 17 3.
Contrastive Learning Framework. 3D CNN Base Encoder. Noise Contrastive Estimation Loss. Fusion signals from multiple camera view-modality combinations 22 3.
Some proposed improvements on the Contrastive Learning Framework 24 3. Noise Contrastive Estimation Loss in weighted combination with Cross Entropy Loss. Enhanced the 3D CNN Base Encoder with Cross Stage Partial technique. Data imballancing alleviation.
Experiments and Evaluation 29 4. Driver Anomaly Detection (DAD) subdataset. Driver & Act dataset. Experiment Results and Evaluation.
The baseline model capability on the sub-DAD dataset. Results of the proposed loss. Results of the base encoder enhancement by applying Cross Stage Partial technique. Results of proposed data sampling setting.
Results of combining all proposed techniques. Conclusion and Future Works 41 5. Summary of Works and Findings. Public datasets mentioned in the thesis 44 List of Tables 1.
Data imbalance problems in many driver monitoring datasets. Mentioned classical machine learning methods using non-visual features 9 2. Mentioned classical machine learning methods using eye (visual) features 11 2. Mentioned classical methods using mouth (visual) features.
Mentioned deep learning methods using visual features. Performance comparison between my baseline model and the best model introduced in [1]. Label smoothing ε value for Cross Entropy. Results of three CSP fusion strategies on the Base encoder.
Results of data sampling strategies on the model. Results of combining all positive methods. The survey of public datasets is mentioned in Chapter 2 of this thesis. 46 v List of Figures 1.
Views & modalities of cameras inside the cabin. Figure in the top left: front depth, in the top right: front IR, in the bottom left: top depth, in the bottom right: top IR [2]. Main components of CLF in training time. A Residual block of ResNet.
Temporal transformation method for creating training data, with the coefficient k = 2, introduced in [1]. The architecture of the CLF system in testing time. Three types of fusion strategies for Cross Stage Partial technique. Improve temporal transformation for the data by using two different stepsizes for normal and abnormal data.
Two examples of normal and abnormal data in the sub-DAD dataset. Driving states statistics on the sub-DAD dataset by minutes. Statistics on training set & testing set by minutes over the sub-DAD dataset. The proportions of training set & testing set by minutes over the sub- DAD dataset.
Example images from Driver & Act dataset at all views and modalities [3]. Area under the ROC curve (AUC) of two different models. AUC results with various β value settings in CENCE loss function. 37 Abbreviation Table No.
Abbreviation Definition 1 AI Artificial intelligence 2 ADAS Advanced Driver Assistance Systems 3 DAD Driver Anomaly Detection 4 CNN Convolutional Neural Network 5 PERCLOS Percentage of Eye Closure 6 MLP Multilayer Perceptron 7 CLF Contrastive Learning Framework 8 DL Deeplearning 9 AUC Area under the ROC curve 10 CE Cross Entropy 11 NCE Noise Contrastive Estimation Abstract Analyzing driver behaviors is crucial for enhancing road safety and preventing accidents. For many years, some classical machine learning methods [4, 5, 6, 2] as well as deep learning methods [7, 8, 9] for this problem were introduced. Although getting remarkable results, these methods only focused on a limited group of the driver’s aspects such as facial states, driving styles, and car conditions instead of the whole context, therefore, they are less reliable in real-world problems. To overcome the limitations of these approaches, a recent study called Contrastive Learning Framework (CLF) [1] was introduced.
By using rich contextual features of drivers from various camera views and modalities, combined with a robust Convolution Neural Network architecture, CLF achieved prominent results on the Driver Anomaly Detection dataset (DAD) [1]. Based on CLF, the thesis tried to enhance CLF’s capabilities by introducing several improvements. Firstly, the thesis reviewed the architecture and key points of the CLF in particular. Besides, some advanced techniques in CNN and training strategy in general were also introduced.
Secondly, the thesis proposed some improvements to the CLF. These improvements included layer optimizations at the base encoder, a new weighted loss combination, and a data sampling strategy during training. We observed that, by applying these changes, the capabilities of the CLF could be significantly improved on a DAD sub-dataset without sacrificing its speed which is very important for such a driver monitoring system (the reason for using the sub-DAD dataset instead of the DAD dataset will be described in more detail in this thesis). Introduction In recent years, with the emergence of new technologies and rapid growth of hardware computing, driver behavior analysis systems have gradually become important components inside cars, contributing a critical role in preventing accidents and ensuring road safety.
In these systems, tracking and identifying the driver’s distraction behavior correctly in real time to alert immediately to the driver is a must-have feature to prevent potential accidents. The reason for this is that if the prediction is not accurate, any error will be able to cause devastating outcomes while if the inference speed is not fast enough, the time when the system alerts may be too late.1 Objective To address the driver behavior analysis problem effectively, many classical machine learning approaches and deep learning approaches were introduced and achieved remarkable results. However, these approaches only focused on a handful number of driver’s features including face, driving style, or vehicle conditions such as speed, steering wheel, etc. while lacking much information from other features in the context as a whole, hence, they may not work too well in real scenarios sometimes.
Recently, a deep learning method, namely Contrastive Learning Framework (CLF), was introduced [1] which can be a step forward in addressing the driver analyzing problems. It achieved 1 prominent results on the Driver Anomaly Detection (DAD) [1] dataset, at above 0.96 in the AUC metric. Inspired by the success of the CLF, the thesis proposed and conducted some modifications on it to have a better understanding of the impact of every component on the overall result as well as try to improve its performance if possible. The final objective is to develop a better variant of CLF with more accurate prediction while keeping the real-time inferencing speed which is vital for driver behavior analysis systems.2 Challenges To develop a reliable deep learning model for analyzing driver behavior, several challenges may need to be addressed: Data imbalance: Data imbalance is a common issue in deep learning and machine learning, where one class of data significantly outnumbers another.
This can lead to models that perform well in the majority class but poorly in the minority class. In driving scenarios, abnormal behaviors such as drowsiness, eating, reading, etc are rare compared with normal driving behaviors. For example, in the DAD dataset [1], the number of normal data recorded was 550 minutes whereas the number of abnormal data was only one-fifth of that, at 100 minutes although these abnormal behaviors may be more important and must be focused. Another kind of imbalance problem is the unequal distribution of recorded data between participants.
For instance, the MPIIGaze [10] dataset comprises over 213 thousand images from 15 participants but the data distribution skewed to only some participants in it. A participant has only about 1.4 thousand images in the dataset whereas it is over 35 thousand images from another participant. These problems require efficient ways of processing data and sophisticated training techniques to address.1) indicates the data imbalance problems on several driver monitoring datasets. 2 Dataset Year Size Imbalance description DAD [4] 2020 95GB 550 minutes are recorded for normal driving and 100 minutes recorded for abnormal driving in the training set.119 million There are a total number of 5.119 million frames, but only a small number of that is the critical events (e.
Fatigue is nearly 1 thousand frames, cellphone use is several dozen frames, etc.6 million Only 2% of the data is taking-off sunglasses class whereas watch video class accounts for 23%.000 Images collected by participants varied from over 1.: Data imbalance problems in many driver monitoring datasets Real-time predicting: In general, the average time the human brain processes and responds to a stimulus is nearly 0. If a driver drives with a speed of 100km/h, it means that his car can move a distance of nearly 28m in a second. Assuming that the monitoring system can only detect the abnormal state of the driver and alert him after 1 second, that means the driver can only be aware of the situation after 1.3 seconds or after a distance of 1. In many cases, it may be too late for the driver to handle if an unexpected thing happens.
That is why real-time predicting of driver behaviors is crucial for the proactive prevention of accidents, enabling vehicles and drivers to respond dynamically to changing conditions. This requires developing an efficient model with low latency but still making sure it’s prediction reliably.: Views & modalities of cameras inside the cabin. Figure in the top left: front depth, in the top right: front IR, in the bottom left: top depth, in the bottom right: top IR [2].3 Contributions The CLF comprises three main components: a Base encoder, a Projection head, and a Noise Contrastive Estimation (NCE) loss. The system consumes the visual signals from multiple cameras with different views and modalities inside the cabin and distinguishes normal driving states from abnormal behaviors (e.
sleeping, reading, or messaging).1) presents the views and modalities of all the cameras inside the cabin. From the design of CLF, this thesis presents some improvements to the Contrastive Learning Framework for the driver monitoring problem. The main contributions of this thesis include: 4 1. Based on the current NCE loss function, the thesis considered another loss combination between NCE Loss and Cross Entropy Loss.
On CE loss the thesis proposed a separate Projection head dedicated to it. From these proposals, the thesis investigated the compatibilities of Noise Contrastive Estimation loss and Cross Entropy loss. The thesis showed that the combination of the two losses brought a significant improvement in the capability of the model.