VIETNAM NATIONAL UNIVERSITY HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY FACULTY OF COMPUTER SCIENCE AND ENGINEERING —————————————– GRADUATION THESIS Using Retrieval Augmentation and Deep Generative Models to Build Question Answering Systems THESIS COMMITTEE: COMPUTER SCIENCE 3 SUPERVISOR: ASSOC. PHAM TRAN VU REVIEWER: DR. LE THANH VAN ——— STUDENT: NGUYEN KHAC HAO - 1852346 HO CHI MINH CITY, JANUARY 2023 Declaration Of Authenticity I declare that the research for this thesis is my own work, conducted under the su- pervision and guidance of Assoc. Pham Tran Vu.
The result of this research is legitimate and has not been published in any form prior to this. All materials used within this research are collected by myself by various sources and are appropriately listed in the references section. In any case of plagiarism, I stand by my actions and will be responsible for it. Ho Chi Minh City University of Technology therefore is not responsible for any copyright infringements conducted within my research.
i Acknowledgement I would like to thank my instructor, Assoc. Pham Tran Vu, not only for his academic guidance and assistance, but also for his patience and personal support which made me truly grateful. I also express my gratitude to Ho Chi Minh City University of Technology for giving me the opportunity to work on this research. Finally, I would like to express my deep sense of gratitude to the research teams and individuals working in computer science who made their incredible work publicly accessi- ble.
Open research papers, datasets and tools are the fundamental elements that helped made this thesis. ii Abstract The recent developments in information technology has given rise to a new generation of conversational applications. A large number of these applications are question answer- ing systems, where the user can ask an information-seeking question, and the application will reply with the corresponding information. Realizing the growth in this kind of application, I set out to build a general-purpose Vietnamese dialogue system that can be quickly adapted into any domain, reducing the amount of development work needed to build a question answering application.
In this work, I introduce a Vietnamese retrieval-augmented question answering system, and ex- plore ways to improve question answering accuracy in Vietnamese. I also introduce 5 new Vietnamese question answering datasets, created using machine translation and two question answering applications. iii Contents 1 Introduction 1 1.1 High computational complexity of deep language models .2 Cost of training deep language models .3 Limited amount of Vietnamese training data .3 Pre-training data .4 Pre-training objective .3 Pre-training data .4 Pre-training objective .3 Pre-training data .4 Pre-training objective .1 Architecture and pre-training .1 Reading comprehension question answering .2 Closed-book question answering .3 Closed-domain question answering .4 Open-domain question answering .1 Creating Vietnamese Question Answering Datasets .2 Reading Comprehension Model .1 Experimenting with fine-tuning data mixtures .2 Further analysis on multilingual fine-tuning .3 Combining best strategies .4 Model experiment 1: BARTpho VQA .5 Model experiment 2: mBERT VQA .6 Model experiment 3: XLM-R VQA .3 Retrieval-Augmentation System .1 Academic Counselling System .1 Vietnamese Question Answering Accuracy .2 English Question Answering Accuracy .3 Answer Retrieval Accuracy .1 Mean reciprocal rank .4 Real-life Testing. 53 6 Conclusion and Future Development 54 6.
55 List of Tables 1.1 Structure of this thesis.1 Detokenized and case-sensitive ROUGE evaluation score for BARTpho and mBART, in Vietnamese text summarization task.2 BLEU scores on PhoMT’s validation set.3 Primary differences between mT5 and other pre-trained multilingual lan- guage models.1 Statistics of the training set of the new Vietnamese question answering dataset. Extractive samples are samples where the answer is a span of text extracted from the document, while abstractive samples are samples where the answer is not an extracted text span from the document.2 Evaluation scores for each model in full-data multilingual training experiment.3 F1 scores (%) of Vi and En+Vi models, evaluated on different data splits.4 English question answering scores for BARTpho En and BARTpho En+Vi.1 Evaluation scores on Vietnamese question answering for baseline models, BARTpho VQA, mBERT VQA and mT5. Scores for mT5 models taken from [16].2 English question answering scores for XLM-R VQA and mT5. Scores for mT5 models taken from [16].3 Mean reciprocal rank scores for answer retrieval of the academic counselling system.4 Exact match scores for answer retrieval of the academic counselling system.
53 vi List of Figures 1.1 Example of question answering system: e-commerce chatbots.2 The linear growth in number of parameters of language models in recent years.1 The architecture of a Transformer model.2 The encoder of the Transformer model.3 The decoder of the Transformer model.4 F1 scores of best models for question answering to date, such as XL-Net [5] and LUKE [6], which are Transformer model.5 An illustration of BERT and mBERT during pre-training.6 Input representation of BERT and mBERT.7 Amount of data in GiB (log-scale) for the 88 languages that appear in both the Wiki-100 corpus used to pre-train mBERT, and the CommonCrawl-100 corpus used to pre-train XLM-R.8 Illustration of the MLM objective used to pre-train XLM-R and XLM.9 Recall@100 scores for unsupervised retrieval.10 An illustration comparing (a) black-box language models and (b) retrieval- augmented systems [17].11 RAG [20], an example implementation or retrieval-augmented model.12 Example of reading comprehension question answering input and output.13 Example of closed-book question answering input and output.1 A diagram of the steps performed during translation of English datasets into Vietnamese.2 Example of HTML tags and whitespaces removal during preprocessing.3 Example of answer span extraction on the SQuAD dataset during prepro- cessing.4 Example of neural machine translation of a preprocessed English data sample.5 Examples of the edits made during translation post-processing.6 Example of a document - question - answer data sample from the final dataset.7 Example of input and output in the multilingual training strategy.8 An example of input and output in the same-dataset multitask training strategy.9 An example of input and output in the multi-dataset multitask training strategy.10 F1 scores for each models trained with different data mixtures, after each training epoch.11 Examples where BARTpho En+Vi answered correctly while BARTpho Vi answered incorrectly.12 F1 scores of Vi and En+Vi models, evaluated on different data splits.13 F1 score of models trained with different data mixtures. NQA is short for NarrativeQA.14 A diagram of the answer span prediction head.15 A diagram of the answer extraction process in encoder-only question an- swering models.16 Architecture of the basic retrieval-augmented question answering system.17 A diagram of the relevance score calculation process in the Retriever.18 An example of the Generator’s input and output.1 Use-case diagram for the user and administrator of the Academic Coun- selling system.2 A diagram of the architecture of the academic counselling application.3 Format of an academic document in the Academic Counselling System.4 Example of Academic Counselling System answering questions by reading academic documents and generating an answer.5 Example of Academic Counselling System answering questions related to transfers.6 Example of Academic Counselling System answering questions related to courses and scheduling.7 Example of Academic Counselling System answering questions related to grading.8 Example of alternative answers in Academic Counselling System.1 Examples of the retrieval testing data for the Academic Counselling Sys- tem. The data is collected from real-life frequently asked questions.1 Problem Statement In the recent years, businesses and organizations have been increasingly using dialogue systems to communicate with their users. Some examples of these systems are e-commerce chatbots, counselling systems and health assistant.
Fundamentally, these are question- answering systems that work to find the right information to respond to the user’s queries. With the increasing use of these systems, there is an increasing need for a general-purpose question answering system that can be quickly adapted into any domain and application, saving development time.1: Example of question answering system: e-commerce chatbots. With deep language models, we can build more flexible, general question-answering systems. However, there is currently a number of problems and limitations that engineers need to overcome before they can bring deep question-answering systems into mainstream use.1 High computational complexity of deep language models Figure 1.2: The linear growth in number of parameters of language models in recent years.
Deep language model’s memory use and processing time increases very quickly with the input size. Language models such as T5 [1] has its memory use quadrupled when doubling the input sequence length. This limits developers from giving deep language models a large data input. For systems such as e-commerce chatbots, where accessing a large company database is required, deep language models become an unsuitable solution.2 Cost of training deep language models Depending on the size of the language model, training can be expensive or unfeasible.
Some implementations today use generative language models that answer user’s queries using knowledge stored within their parameters, learned during training time. When the information becomes outdated, the only way to update these models is to train them again. The cost of developing a language model from scratch and maintenance makes them unsuitable for real-life use.3 Limited amount of Vietnamese training data Deep language models requires a large amount of data for training and fine-tuning. Currently, there is not as many Vietnamese datasets for training language models as English.
For question answering, there is no large-scale, publicly-accessible Vietnamese dataset available. The limited amount of Vietnamese dataset available is one of the rea- sons traditional, specific-domain question answering systems is still currently the prefer- 2 able choice.2 Objective The objective of this work is to create a Vietnamese retrieval-augmented question answering system. This system can be quickly adapted to specific domains, reducing engineering work. When adapting this system to a new domain, no training is needed.
And when developers need to update the application’s data, they can update the data files without re-training the language model. This work is also a study on how to to apply deep language models in Vietnamese ap- plications. In this work, I explore with creating new dataset, new data mixture strategies and new Vietnamese question answering models.1 Academic significance • The new Vietnamese question answering datasets and question answering models introduced in this work can become useful resources for future studies in Vietnamese question answering. • The optimal data mixture experiment in this work can be useful for future studies.2 Practical significance • The retrieval-augmentation question answering system can be quickly modified to create applications for specific use, such as counselling systems and customer support systems.
This can help reduce development time. I demonstrated this significance by implementing two applications using this system. • Vietnamese question answering models and datasets introduced in this work can help other developments in bringing deep language models to real-life applications.4 Scope Currently, there are many types of dialogue system with different capabilities. There are question answering systems that can work with images, question answering systems that can do reasoning and calculations, and personal assistants that can accomplish tasks such as sending messages and e-mails.
The scope of this topic is limited to single-turn question answering systems that finds the information for the user. This system can work with information sources of any domain, but cannot do reasoning, calculations or accomplishing tasks other than question answering.5 Thesis Structure There are 6 chapters in this thesis: Chapter Content 1 Introduction to the objectives of this thesis Theoretical background as foundation knowledge of the algorithms, 2 models and techniques used in this work. The design, experiment and implementation process for the objectives 3 of this work. 4 Applying the results of this work into real-life applications.
Evaluations and comparisons of the results of this work to other similar 5 work. 6 Conclusions, results of this thesis and future plans.1: Structure of this thesis. 4 Chapter 2 Theoretical Background 2.1: The architecture of a Transformer model. The Transformer model [2] is a deep language model proposed in June 2017.
Trans- former adopts the mechanism of self-attention, learning from not only individual words but also the context of each word. The additional training parallelization allows training on larger datasets than was once possible. Since the introduction of the Transformer 5 model, many variations based on this model have been introduced.