A STUDY ON MACHINE TRANSLATION FOR LOW-RESOURCE LANGUAGES By TRIEU, LONG HAI submitted to Japan Advanced Institute of Science and Technology, in partial fulfillment of the requirements for the degree of Doctor of Philosophy Written under the direction of Associate Professor Nguyen Minh Le September, 2017 A STUDY ON MACHINE TRANSLATION FOR LOW-RESOURCE LANGUAGES By TRIEU, LONG HAI (1420211) A thesis submitted to School of Information Science, Japan Advanced Institute of Science and Technology, in partial fulfillment of the requirements for the degree of Doctor of Information Science Graduate Program in Information Science Written under the direction of Associate Professor Nguyen Minh Le and approved by Associate Professor Nguyen Minh Le Professor Satoshi Tojo Professor Hiroyuki Iida Associate Professor Kiyoaki Shirai Associate Professor Ittoo Ashwin July, 2017 (Submitted) Copyright c 2017 by TRIEU, LONG HAI Acknowledgements Abstract Current state-of-the-art machine translation methods are neural machine translation and statistical machine translation, which based on translated texts (bilingual corpora) to learn translation rules automatically. Nevertheless, large bilingual corpora are unavailable for most languages in the world, called low-resource languages, that cause a bottleneck for machine translation (MT). Therefore, improving MT on low-resource languages becomes one of the essential tasks in MT currently. In this dissertation, I present my proposed methods to improve MT on low-resource languages by two strategies: building bilingual corpora to enlarge training data for MT systems and exploiting existing bilingual corpora by using pivot methods.
For the first strategy, I proposed a method to improve sentence alignment based on word similarity learnt from monolingual data to build bilingual corpora. Then, a multilingual parallel corpus was built using the proposed method to improve MT on several Southeast Asian low-resource languages. Experimental results showed the effectiveness of the proposed alignment method to improve sentence alignment and the contribution of the extracted corpus to improve MT performance. For the second strategy, I proposed two methods based on semantic similarity and using grammatical and morphological knowledge to im- prove conventional pivot methods, which generate source-target phrase translation using pivot language(s) as the bridge from source-pivot and pivot-target bilingual corpora.
I con- ducted experiments on low-resource language pairs such as the translation from Japanese, Malay, Indonesian, and Filipino to Vietnamese and achieved promising results and im- provement. Additionally, a hybrid model was introduced that combines the two strategies to further exploit additional data to improve MT performance. Experiments were con- ducted on several language pairs: Japanese-Vietnamese, Indonesian-Vietnamese, Malay- Vietnamese, and Turkish-English, and achieved a significant improvement. In addition, I utilized and investigated neural machine translation (NMT), the state-of-the-art method in machine translation that has been proposed currently, for low-resource languages.
I compared NMT with phrase-based methods on low-resource settings, and investigated how the low-resource data affects the two methods. The results are useful for further de- velopment of NMT on low-resource languages. I conclude with how my work contributes to current MT research especially for low-resource languages and enhances the development of MT on such languages in the future. Keywords: machine translation, phrase-based machine translation, neural-based ma- chine translation, low-resource languages, bilingual corpora, pivot translation, sentence alignment 2 Acknowledgements For three years working on this topic, it is my first long journey that attract me to the academic area.
It is also one of the biggest challenges that I have ever dealt with. This work gives me a lot of interesting knowledge and experiences as well as difficulties that require me with the best efforts. At the moment of writing this dissertation as a summary for the PhD journey, it reminds me a lot of support from many people. This work cannot be completed without their support.
First of all, I would like to thank my supervisor, Associate Professor Nguyen Minh Le. Professor Nguyen gives me a lot of comments, advices, discussions in my whole three-year journey from the starting point when I approached this topic without any prior knowledge about machine translation until my last tasks to complete my dissertation and research. Doing PhD is one of the most interesting things in studying, but it is also one of the most challenge things for everyone in the academic career. Thanks to the useful and interesting discussions with professor Nguyen, I have overcome the most difficult periods in doing this research.
Not only teach me some first lessons and skills in doing research, professor Nguyen also has interesting and useful discussions that help me a lot in both studying and the life. I would like to thank the committee: Professor Satoshi Tojo, Professor Hiroyuki Iida, Associate Professor Ittoo Ashwin, Associate Professor Kiyoaki Shirai for their comments. This can be one of the first work in my academic career, that cannot avoid a lot of mistakes and weaknesses. By discussing with the professors in the committee, and receiving their valuable comments, they help me a lot in improving this dissertation.
I also would like to thank my collaborators: Associate Professor Nguyen Phuong Thai for his comments, advices, and experience in sentence alignment and machine translation. I would like to thank Vu Tran, Tin Pham, Viet-Anh Phan for their interesting discussions and collaborations in doing some topics in this research. Thanks so much to Vu Tran, Chien Tran for their technical support. I would like to thank my colleagues and friends, Truong Nguyen, Huy Nguyen, for their support and encourage.
I also would like to give a special thank to professor Jean- Christophe Terrillon Georges for his advices and comments on the writing skills and En- glish manuscripts of my papers, special thank to professor Ho Tu Bao for valuable advices in research. Thanks so much to Danilo S. Carvalho, Tien Nguyen for their comments. Last but not least, I would like to thank my parents, Thi Trieu, Phuong Hoang, my sister Ly Trieu, and my wife Xuan Dam for their support and encouragement in all time not only in this work but in my life.
3 4 Table of Contents Abstract 1 Acknowledgements 1 Table of Contents 3 List of Figures 4 List of Tables 6 1 Introduction 7 1.2 MT for Low-Resource Languages .1 Statistical Machine Translation .1 Phrase-based SMT .1 Length-Based Methods .2 Word-Based Methods .3 Triangulation: The Representative Approach in Pivot Methods .4 Neural Machine Translation. 19 3 Building Bilingual Corpora 21 3.1 Dealing with Out-Of-Vocabulary Problem .1 Word Similarity Models. 22 1 TABLE OF CONTENTS 3.2 Improving Sentence Alignment Using Word Similarity .2 Building A Multilingual Parallel Corpus .5 Experiments on Machine Translation. 40 4 Pivoting Bilingual Corpora 41 4.1 Semantic Similarity for Pivot Translation .1 Semantic Similarity Models .2 Semantic Similarity for Triangulation .3 Experiments on Japanese-Vietnamese .4 Experiments on Southeast Asian Languages .2 Grammatical and Morphological Knowledge for Pivot Translation .1 Grammatical and Morphological Knowledge .2 Combining Features to Pivot Translation .1 Using Other Languages for Pivot .2 Rectangulation for Phrase Pivot Translation.
70 5 Combining Additional Resources to Enhance SMT for Low-Resource Languages 72 5.1 Enhancing Low-Resource SMT by Combining Additional Resources .2 Experiments on Japanese-Vietnamese .3 Experiments on Southeast Asian Languages .4 Experiments on Turkish-English .1 Exploiting Informative Vocabulary. 82 2 TABLE OF CONTENTS 5. 86 6 Neural Machine Translation for Low-Resource Languages 88 6.1 Neural Machine Translation .2 Byte-pair Encoding .2 Phrase-based versus Neural-based Machine Translation on Low-Resource Languages. NMT on Low-Resource Settings .3 Improving SMT and NMT Using Comparable Data .3 A Discussion on Transfer Learning for Low- Resource Neural Machine Translation.
95 7 Conclusion 96 3 List of Figures 2.1 Pivot alignment induction .2 Recurrent architecture in neural machine translation .1 Word similarity for sentence alignment .2 Experimental results on the development and test sets .3 SMT vs NMT in using the Wikipedia corpus .1 Semantic similarity for pivot translation .2 Pivoting using syntactic information .3 Pivoting using morphological information .1 A combined model for SMT on low-resource languages. 73 4 List of Tables 3.1 English-Vietnamese sentence alignment test data set .2 IWSLT15 corpus for training word alignment .3 English-Vietnamese alignment results .4 Sample English word similarity .5 Sample Vietnamese word similarity .6 OOV ratio in sentence alignment .7 Sample English-Vietnamese alignment .8 English word similarity .9 Sample IBM Model 1 .10 Induced word alignment .11 Wikipedia database dumps’ resources used to extract parallel titles .12 Extracted and processed data from parallel titles .13 Sentence alignment output .14 Extracted Southeast Asian multilingual parallel corpus .15 Monolingual data sets .16 Experimental results on the development and test sets .17 Data sets on the IWSLT 2015 experiments .18 Experimental results using phrase-based statistical machine translation .19 Experimental results on neural machine translation .20 Comparison with other systems participated in the IWSLT 2015 shared task 40 4.1 Bilingual corpora for Japanese-Vietnamese pivot translation .2 Japanese-Vietnamese development and test sets .3 Monolingual data sets of Japanese, English, Vietnamese .4 Japanese-Vietnamese pivot translation results .5 Bilingual corpora of Southeast Asian language pairs .6 Bilingual corpora for pivot translation of Southeast Asian language pairs .7 Monolingual data sets of Indonesian, Malay, and Filipino .8 Pivot translation results of Southeast Asian language pairs .9 Examples of grammatical information for pivot translation .10 Southeast Asian bilingual corpora for training factored models .11 Results of using POS and lemma forms .12 Indonesian-Vietnamese results .13 Filipino-Vietnamese results. 55 5 LIST OF TABLES 4.14 Input factored phrase tables .15 Extracted phrase pairs by triangulation .16 Out-Of-Vocabulary ratio .17 Results of statistical significance tests .18 Experimental results on different metrics: BLEU, TER, METEOR .19 Ranks on different metrics .20 Spearman rank correlation between metrics .21 Wilcoxon on Malay-Vietnamese .22 Wilcoxon on Indonesian-Vietnamese .23 Wilcoxon on Filipino-Vietnamese .24 Wilcoxon on Malay-Vietnamese .25 Wilcoxon on Indonesian-Vietnamese .26 Wilcoxon on Filipino-Vietnamese .27 Sample translations: POS and lemma factors for pivot translation .28 Sample translation: Indonesian-Vietnamese .29 Sample translation: Filipino-Vietnamese .30 Using other languages for pivot .31 Using rectangulation for phrase pivot translation .1 Japanese-Vietnamese results on the direct model .2 Japanese-Vietnamese results on the combined models .3 Results of Japanese-Vietnamese on the big test set .4 Results of statistical significance tests on Japanese-Vietnamese .5 Southeast Asian results on the direct models .6 Southeast Asian results on the combined model .7 Bilingual corpora for Turkish-English pivot translation .8 Experimental results on the Turkish-English .9 Experimental results on the English-Turkish translation .10 Building a bilingual corpus of Turkish-English from Wikipedia .11 Dealing with out of vocabulary problem using the combined model .12 Sample translations: using the combined model (Japanese-Vietnamese) .14 Sample translations: using the combined model (Filipino-Vietnamese) .1 Bilingual data set of Japanese-English .2 Experimental results in Japanese-English translation .3 Bilingual data sets of Indonesian-Vietnamese .4 Experimental results on Indonesian-Vietnamese translation .5 Experimental results English-Vietnamese .6 English-Vietnamese results using the Wikipedia corpus .1 Machine Translation Translation between languages is an important demand of humanity. With the advent of digital computers, it provided a basis for the dream of building machines to translate languages automatically.
Almost as soon as electronic computers appeared, people made efforts to build automatic systems for translation, which also opened a new field: machine translation. As defined in Hutchins and Somers, 1992 [33], machine translation (MT) is "computerized systems responsible for the production of translation from one natural language to another, with or without human assistance". Machine translation has a long history in its development. Various approaches were explored such as: direct translation (using rules to map input to output), transfer methods (analyzing syntactic and morphological information), and interlingual methods (using representations of abstract meaning).
The field attracted a lot of interest from community like: a study of realities of machine translation from US funding agencies in 1966 (ALPAC report), commercial systems from the past (Systran in 1968, Météo in 1976, Logos and METAL in 1980s) to current development by large companies (IBM, Microsoft, Google), and many projects in universities and academic institutes. Dominated approaches of current machine translation are statistical machine translation (SMT) and neural machine translation (NMT), which are based on resources of translated texts, a trend of data-driven methods.