VIETNAM NATIONAL UNIVERSITY - HO CHI MINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY FACULTY OF COMPUTER SCIENCE BACHELOR THESIS Gradual Domain Adaptation in Scene Text Recognition Bachelor of Computer Science (Honors degree) HO CHUNG DUC KHANH- 19520624 NGUYEN THI MINH PHUONG- 19522065 Supervised by PhD. THANH DUC NGO HO CHI MINH CITY, 2023 VIETNAM NATIONAL UNIVERSITY - HO CHI MINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY FACULTY OF COMPUTER SCIENCE BACHELOR THESIS Gradual Domain Adaptation in Scene Text Recognition Bachelor of Computer Science (Honors degree) HO CHUNG DUC KHANH- 19520624 NGUYEN THI MINH PHUONG- 19522065 Supervised by PhD. THANH DUC NGO HO CHI MINH CITY, 2023 COMMITTEE The Thesis Defense Committee has been carefully established in accordance with the Decision 155/QD-DHCNTT, issued on 01/03/2023 by the President of the University of Information Technology. This committee is comprised of eminent individuals who possess a great deal of expertise and knowledge in the specific area of study that is relevant to the thesis defense.
To ensure that all aspects of the thesis defense are properly addressed, the following personnel have been carefully chosen to comprise the committee: ¢ Chairman: PhD. Le Dinh Duy. Nguyen Thanh Son. Le Minh Hung.
ACKNOWLEDGEMENTS We would like to express our deepest gratitude to all those who have been in- volved in the successful completion of this thesis. First and foremost, we would like to thank our supervisor, PhD. Thanh Duc Ngo, for their invaluable guidance and support throughout the course of this project. Thanh has been an incredible mentor, providing insightful feedback and suggestions, and challenging us to think outside of the box.
we are deeply grate- ful for their commitment and dedication to our project. We are also thankful to all the members of our research group for their help and encouragement. We are especially grateful to Tan and Hung, whose technical guidance and knowledge of domain adaptation techniques have been instrumen- tal in the successful completion of this thesis. We are thankful to the reviewers for their valuable feedback and suggestions, which have been extremely helpful in improving the quality of this thesis.
We would also like to thank our family and friends for their support and encour- agement throughout the entire process. Finally, we would like to express our gratitude to the computing resources pro- vided at MMLAB UIT and the Faculty of Computer Science, which enabled us to develop the algorithms and experiments presented in this thesis. All in all, this project would not have been possible without the assistance and help of the people mentioned above. Our sincere appreciation goes out to each of them.
Contents IAbstractl 1_ Introductioni 1.2 Scene Text Recogniion|.4 Gradual Domain Adaptation|. eee 11 [6 Structure of This Thesisl.1 Language-free methods|.2 Language-based methodsl. Gradual Domain Adaptation]. xa 24 BAL 5yntheicDataseBl.2 Unlabeled Real-world Datasetl.
00000 eee ee 26 3_ Gradual Domain Adaptation in Scene Text Recognition via Pseudo-labeling] 28 3.1 Domain Adaptation in Scene Text Recognition via Pseudo-labeling| 28 3.2 The Scene Text Recognition Framework|.2_ Gradual Domain Adaptation and Domain Routing].1 Unsupervised Domain Adaptation on synthetic-trained model].2 Gradual Domain Adaptation on synthetic-trained model]. 46 [£4 Comparing domain routing approach with state-of-the-art models] 52 54 See 54 ¬ 55 56 List of Figures 1.1 Scene Text Recognition task: output the text content in the image. The sample image is taken from ICDAR 2015 dataset|.2 Common challenges in Scene Text Recognition}.4 Gradual Domain Adaptation for Scene Text Recognition] .1 Hlustration of Scene Text Recognition framework].2 Transformation stage from 2] THỦ: - - - - --- ---------- 15 [2.3 Some samples of three unlabeled real-world đatasets|.1 Overview of Pseudo-labeling approach|.2 Two model combinations according to the STR Framework. (Image [rom Do eee 32 3.
Structure of Spatial Transform Networks.4 Structure of BiLSTM.5 Ilustration of Attention mechanisml.6 The Portraits dataset of students from 1905 to 2005 [17]|.1 Illustration of Experiment Resultsl|.2 Illustration of Experiment ResultslH|. 43 Samples of three unlabeled real-world datasets|.4 Illustration of Experiment ResultsM]. List of Tables 1 Experimental resultsl|.3 Compare three unlabeled real-world datasets| .4 Experiment results on different domain routings}. 51 5 Comparison between our domain routing approach and state-of- the-art methodsl.
53 Abstract Textual information is essential to virtually all aspects of our daily lives, and au- tomating the process of bringing this information onto the digital world is a key goal of research in the field of computer vision. Scene Text Recognition (STR) is a particularly important task in this domain, as it has numerous applications in areas such as automated number plate recognition for vehicles, access control systems, and much more. However, the challenge of Scene Text Recognition is that it requires large amounts of annotated data, which is very expensive and time-consuming to collect. As such, a common approach is to leverage generated synthetic data during train- ing, and test the result on real-world data.
Unfortunately, this approach can be ineffective, as there is often a large discrepancy between the synthetic data used for training and the data in the real world, referred to as a “domain gap”. Recent approaches to Scene Text Recognition have attempted to address the do- main gap by adopting Domain Adaptation techniques, which try to minimize the discrepancy between the two domains in a semi-supervised manner. However, when the gap between the two domains is too large, Domain Adaptation may not be effective. This thesis proposes and evaluates a Gradual Domain Adaptation approach, which trains the model on multiple intermediate domains in order to minimize the gap before training on the final domain.
This technique is evaluated in terms of its ef- fectiveness in reducing the domain gap, as well as the impact of adding interme- diate domains, changing domains, and finding the appropriate domain routing. As a result, the experiments are able to confirm the following: 1. Gradual Domain Adaptation improves Scene Text Recognition baseline per- formance by up to 3.12%, compared to Domain Adaptation approach. 1 List of Tables 2.
Switching domains order improves performance by 1. Performance is consistently improved when adapting with increasingly sta- ble domain routings. We observe a performance boost of up to 5.81% in our experiments. Moreover, our method was able to outperform state-of-the-art approaches by leveraging the domain routing approach.
This demonstrates the potential for this approach in scene text recognition. Keywords: Gradual Domain Adaptation, Domain Routing, Domain Adaptation, Scene Text Recognition, Unsupervised Domain Adaptation, Pseudo-labeling.1 Overview Textual information is essential to virtually all aspects of our daily lives, and au- tomating the process of bringing this information onto the digital world is a key goal of research in the field of computer vision. Scene Text Recognition (STR) is a particularly important task in this domain, as it has numerous applications in areas such as automated number plate recognition for vehicles, access control systems, and much more. However, the challenge of Scene Text Recognition is that it requires large amounts of annotated data, which is very expensive and time-consuming to collect.
As such, a common approach is to leverage generated synthetic data during train- ing, and test the result on real-world data. Unfortunately, this approach can be ineffective, as there is often a large discrepancy between the synthetic data used for training and the testing data in the real world, referred to as a “domain shift”. Recent approaches to Scene Text Recognition have attempted to address the do- main shift by adopting Domain Adaptation techniques, which try to minimize the discrepancy between the two domains in a semi-supervised manner. How- ever, when the gap between the two domains is too large, Domain Adaptation 3 Chapter 1.
Introduction may not be effective. This thesis proposes and evaluates a Gradual Domain Adaptation approach, which trains the model on multiple intermediate domains in order to minimize the gap before training on the final domain. This technique is evaluated in terms of its ef- fectiveness in reducing the domain gap, as well as the impact of adding interme- diate domains, changing domains, and finding the appropriate domain routing. The main research questions of this thesis are: 1.
What is the performance of Gradual Domain Adaptation in Scene Text Recog- nition? 2. Does intermediate domain routings affect overall performance of Scene Text Recognition model using Gradual Domain Adaptation? 3. How to choose a good domain routing when applying Gradual Domain Adaptation for Scene Text Recognition? 1.2 Scene Text Recognition Scene Text Recognition (STR) is an important task in the fields of computer vision and natural language processing, and has been heavily studied due to its many useful applications. Scene Text Recognition is capable of recognizing text com- ponents in a wide range of settings, from street signs and license plates, to news- paper headlines, advertisements, and digital images and videos.
This powerful technology enables tasks that would otherwise be impossible, such as quickly and accurately searching, translating, and verifying documents. In addition to aiding law enforcement, Scene Text Recognition can also be used to help compa- nies read customer reviews and detect text in security documents for automated Chapter 1. It is also a powerful tool for a variety of industries, from automotive to healthcare, and even retail, as it can be used to build more efficient and secure systems. Scene Text Recognition is a versatile technology that has numerous ap- plications in multiple fields and can be used to greatly improve the speed and accuracy of various tasks, making it an invaluable tool.
The input of this task is an image containing a text instance and the output of the task is the corresponding text sequence. As an illustration, figure [I-1]shows an example of the input and output of the Scene Text Recognition task. Specif- ically, the image contains the text "MOVING" which is the input of the Scene Text Recognition task. The corresponding output of the task is the text sequence "MOVING" which can be seen in the output box.
By recognizing the text compo- nents present in a scene, the Scene Text Recognition task can be used to perform a range of tasks from automating document processing to providing assistance for the visually impaired. INPUT OUTPUT FIGURE 1.1: Scene Text Recognition task: output the text content in the image. The sample image is taken from ICDAR 2015 dataset. Despite the progress made on Scene Text Recognition, the task still faces several challenges.
A key challenge is the diversity of the conditions of the input images. The text components can be in various fonts, sizes, colors, orientations, and even shapes. This means that a single recognition algorithm may not work optimally Chapter 1. Introduction across different types of images.
Additionally, the background can contain noise, complex patterns and other distracting elements, making the task more difficult. For instance, figure [1.2] demonstrates some of the common challenges of Scene Text Recognition, such as irregular fonts (Fig. noisy background (Fig. irregular text orientation (Fig.3cp, and uneven lighting/obstructed texts (Fig.
These conditions make it difficult to process the images correctly, as the system must take into account the context of the image in order to accurately recognize the text components. As a result, a robust and accurate Scene Text Recognition algorithm must be able to handle possible variations in the input images. An additional challenge lies in the limited amount of annotated data available for training. To address this issue, researchers often resort to the use of synthetic data to train their models, since it is possible to generate large amounts of data in this manner [29] [20].
While this approach can be successful in some cases, the models trained on synthetic data tend to have poor performance when applied to real-world data due to the domain gap To reduce this domain gap and increase the accuracy of the models, recent works have adopted domain adaptation techniques to bridge the difference between synthetic and real-world data 1. Domain Adaptation Domain Adaptation (DA) is a powerful machine learning technique for bridging the gap between two different datasets, particularly in cases where the training and testing data have different distributions (Fig.