NGUYEN MINH LY- 20521592 KHOA LUAN TOT NGHIEP ĐÁNH GIA HIEU QUA CUA KIÊN TRÚC CNN- TRANSFORMERS TRONG VIEC PHAN LOAI TINH TRANG HU HONG DUONG BO (EVALUATING THE EFFECTIVENESS OF CNN-TRANSFORMERS ARCHITECTURE IN CLASSIFYING ROAD DAMAGE CONDITIONS) CU NHAN NGANH KHOA HOC MAY TINH GIANG VIEN HUONG DAN TS. LE KIM HUNG TP. HO CHI MINH, 2024 Dissertation Defense Committee List The dissertation defense committee, established according to Decision No. 154/QD-DHCNTT dated March 1, 2023, by the Rector of the University of Information Technology.
Acknowledgements First of all, I would like to express my sincere gratitude to all the professors and teachers working and teaching at the University of Information Technology - VNU- HCM for the knowledge, lessons, and valuable experiences that I have acquired during my recent journey. I wish the Department of Computer Science in particular, and the University of Information Technology - VNU-HCM in general, continued brilliant success in the field of education, always training talents for the country, and remaining a firm and attractive destination for future generations of students. I would also like to extend my heartfelt thanks to Dr. Le Kim Hung.
Thanks to his experiences, lessons, care, assistance, and guidance, I have overcome the difficulties and challenges in the process of completing this graduation thesis. Next, I would like to express my sincere thanks to my family for always believing in and encouraging me throughout my studies at the University of Information Technology - VNU-HCM, giving me additional motivation to strive for development and achieve the success I have today. Finally, I would like to thank my fellow students at the University of Information Technology - VNU-HCM for their companionship, assistance, enthusiasm in sharing opinions, and suggestions to help improve and perfect my graduation thesis. Ho Chi Minh, 2024 Table of Contents Acknowledgements 4 ABSTRACT 11 CHAPTER 1.
Objectives of the Thesis 14 1. Subjects and Scope of Research 14 1. Scope of Research 15 CHAPTER 2. Overview of Artificial Intelligence, Machine Learning, Deep Learning 16 2.
Typical Architectures in Deep Learning 21 2. Convolution Neural Network 25 2. Image classification problem 34 2. Some studies on the problem of detecting damaged paths 35 CHAPTER 3.
Overview of the Model 36 3. Zero shot learning 45 CHAPTER 4. EXPERIMENTS AND EVALUATION 50 4. Experimental results 59 CHAPTER 5.
SUMMARY AND DEVELOPMENT DIRECTIONS 69 5. The achieved results 69 5. Development Directions 70 REFERENCES 72 List of images Figure 1. Investigate the overall management theory, applications, and classifications in the fields of AI, ML, and DL.The performance differences among various AI and ML model groups.
Transformers arChIf€CfUT€.Proposed model architecture. The skip connection architecture was invented in resnet. Architecture of the ResNet50 model.---- -5- -sc+c+csereersee 40 Figure 7. Ways to scale up neural network architectures.
The baseline architecture of the EfficientNet model. Compare the accuracy of the models on the ImageNet dataset. The architecture of the EfficientNetB3 model. Image classification with Zero-shot learning.
CLIP training process on 400 million image-caption data pairs. The process of predicting the label of an image using the clip model Figure 14. Distribute data in train, valid and test S€fs. The ratio of images between the classes in the dataset; 0: non-damaged road class; 1: damaged road CÏaSS.- - -- s91 ng ng c 52 Figure 16.
Distribution of images for each country in the train da(a. Distribution of images for each country in the valid data. Distribution of images for each country in the test dafa. Some images from the dataset.
a, b: Images in the case of undamaged roads; c, d: Images in the case of damaged roads. The loss and accuracy scores of the model during training. The GradCAM results display the important regions on an image that the model uses to make Dr€dICfIOTIS.- 5 E111 19112 1231 11 HH HH ni, 65 List of Tables Table 1. Details the number of images in the đafa.
Values of TP, FN, FP, TN in the confusion matr1X. Result for Zero-shot learning for each cÌaSS€S. The overview result for zero-shot learnIng.- --‹--- «<< «++<ex++es 60 Table 7. The number of parameters for each model.
The detailed accuracy metrics of the proposed model using various TT€8SUT€ITRIES.G- 6 5162112001855 111112 T91 nh HT TH HH TH nh nhờ 64 Table 9. The results of comparing the accuracy of our model ("Ours") with the CNN models 1177777.- óc + 2111911 91 931 9 1 9311 v1 ng nhiệt 68 List of Abbreviations Artificial Intelligence 10 ABSTRACT Detecting road damage is an essential and crucial task for ensuring road infrastructure, traffic safety, and maintaining the supply chain for the economy. With the rapid technological development in recent years, especially in automated image processing techniques in the field of artificial intelligence, this thesis researches solutions for automatically detecting road damages. The aim is to improve accuracy and consider reasonable computational resources for practical deployment, meeting the significant global demand.
In this thesis, I propose a new model architecture that utilizes a combination of convolutional neural network (CNN) and transformers for image classification problems. This architecture leverages the strength of CNNs in extracting features from intermediate to advanced levels of input images, and then processes these through transformer encoder blocks. This process uncovers hidden features in the feature vector that CNNs may not detect, thereby enhancing accuracy for image classification. Moreover, we compile various related datasets to create a comprehensive dataset for evaluating the effectiveness of the proposed solution compared to the best available solutions.
Thesis Title Evaluating the effectiveness of CNN-Transformers architecture in classifying road damage conditions. Problem Statement In the current context, the automatic assessment of road damage is not only a technical issue but also an urgent need for society. Cities and traffic management authorities globally are increasingly recognizing the importance of maintaining road infrastructure. This is crucial not only for traffic safety but also significantly contributes to improving transportation efficiency, especially in the context where road transport accounts for a predominant share of freight transportation in many countries.
A reliable statistic shows that the proportion of goods transported by road in European countries accounted for 77.3% in the year 2021. However, current solutions such as human monitoring through video or image analysis from unmanned aerial vehicles, while partly addressing the issue, face many limitations regarding labor and equipment costs, raising questions about their widespread applicability. Additionally, management agencies often lack technological expertise in deploying and maintaining these advanced and modern systems. Recently, new solutions, although applying advanced technologies like machine learning and automatic image processing, still cannot fully meet these needs.
For example, the crack detection model on roads developed by the University of Tokyo, while achieving high accuracy, requires up to 1500ms to process a single image when experimented on a smartphone. This clearly does not meet the real-time requirements in practical applications [2]. Another solution is the work of Yachao Yuan and colleagues, 12 who attempted to address this issue by combining various simple image processing techniques to increase processing speed [3]. Although the model achieved impressive accuracy and prediction speed, the dataset used was not large and diverse enough to conclusively demonstrate the solution's effectiveness across different road types in various countries.
This highlights a significant limitation in developing models that can generalize effectively across various road types worldwide. Machine Learning (ML) and Artificial Intelligence (AI) have opened a new approach to addressing this issue. The combination of AI with image processing techniques holds great potential in automatically and accurately detecting and classifying road damages. However, developing AI models that can generalize well across various environmental and road conditions remains a significant challenge.
Additionally, processing and analyzing the large volume of collected data is a considerable challenge, requiring a smooth integration of advanced image processing techniques and machine learning. In the field of deep learning in recent years, the emergence of the transformer architecture has revolutionized the field of artificial intelligence, particularly in natural language processing. There have been many studies utilizing the power of transformers to improve accuracy in image processing tasks, achieving significant advancements. The superiority of this approach is due to the attention mechanism, which allows the model to view the entire context rather than just focusing on local areas, as is the case with traditional CNNs.
Therefore, a current trend is to research models that combine both CNNs and transformers to create significant technological advancements. The research and development of new solutions are necessary both technically and socially. Advanced research is needed to develop AI models capable of quickly, accurately, and efficiently classifying road damages at a reasonable cost. This will play a crucial role in improving the safety and efficiency of the global road traffic system.
13 In this context, the topic of this thesis aims not only to solve a technical problem but also to contribute to the sustainable development of global road infrastructure. The research and development of AI-based automatic road damage classification solutions are not only a significant step forward in the technical field but also a practical contribution to improving the quality of life and safety of the community. Objectives of the Thesis e Explore the fields of artificial intelligence, machine learning, deep learning, and their applications. e Research several existing studies and methods in the problem of classifying road damage conditions.
e Focus on designing and testing architectures that combine Convolutional Neural Networks (CNN) and Transformers. The goal is to create an optimal solution for the road damage classification problem, leveraging the advantages of both architectures: the powerful image feature recognition capability of CNNs and the ability to process sequential, complex data of Transformers. e Compare and evaluate the performance of the proposed model against models that use only the CNN architecture. e Evaluate the model's ability to deploy the proposed model on an edge device.
Subjects and Scope of Research 1. Research Subjects This thesis focuses on: 14 Modern network architectures in machine learning and artificial intelligence, including ANN (Artificial Neural Networks), CNN (Convolutional Neural Networks), and Transformers. The problem of image classification for the detection of damaged road surfaces in photographs. The feasibility of deploying these models in practical applications, particularly on devices with limited hardware capabilities.
Scope of Research The scope of research includes: Research on existing solutions to address the problem of road damage classification. Research on effective models in the field of image processing, specifically for image classification problems. Overview of Artificial Intelligence, Machine Learning, Deep Learning @ — — — — — ARTIFICIAL INTELLIGENCE =< A technique which enables machines Artificial Intelligence _ -“ to mimic human behaviour kh Machine Learning MACHINE LEARNING max" ”——————^T—T Subset of AI technique which use \ | statistical methods to enable machines x to improve with experience ~~ DEEP LEARNING ~— — — — — — Subset of ML which make the computation of multi-layer neural network feasible Figure 1. Investigate the overall management theory, applications, and classifications in the fields of AI, ML, and DL.
Artificial Intelligence: In the field of computer science, artificial intelligence, or AI, sometimes referred to as synthetic intelligence, is intelligence exhibited by machines, contrasting with the natural intelligence of humans. Typically, the term "artificial intelligence" is used to describe server (or computer) systems capable of mimicking "cognitive" functions often associated with the human mind, such as "learning" and "problem-solving." [4] Artificial intelligence systems are classified based on their ability to replicate human characteristics and are broadly divided into three main types as follows: 16 e Narrow Artificial Intelligence (ANI): At this level, artificial intelligence can only solve problems in a specialized domain, such as image classification or spam email filtering. e Artificial General Intelligence (AGI): This type of intelligence is similar to human capabilities, meaning it can perform tasks that humans can do and can be considered a miniature representation of human intelligence. e Artificial Super Intelligence (ASI): At this level, artificial intelligence surpasses human intelligence in its capabilities.