VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY VU HUY HIEN BOOTSTRAPPING SMT USING UNANNOTATED CORPORA OF THE SOURCE LANGUAGE MASTER THESIS OF INFORMATION TECHNOLOGY Hanoi - 2014 LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY VU HUY HIEN BOOTSTRAPPING SMT USING UNANNOTATED CORPORA OF THE SOURCE LANGUAGE Major: Computer science Code: 60 48 01 MASTER THESIS OF INFORMATION TECHNOLOGY SUPERVISOR: PhD. Nguyen Phuong Thai Hanoi - 2014 LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com 2 LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com ORIGINALITY STATEMENT ‘I hereby declare that this submission is my own work and to the best of my knowledge it contains no materials previously published or written by another person, or substan- tial proportions of material which have been accepted for the award of any other degree or diploma at University of Engineering and Technology (UET/Coltech) or any other educational institution, except where due acknowledgement is made in the thesis. Any contribution made to the research by others, with whom I have worked at UET/Coltech or elsewhere, is explicitly acknowledged in the thesis. I also declare that the intellectual content of this thesis is the product of my own work, except to the extent that assistance from others in the project’s design and conception or in style, presentation and linguistic expression is acknowledged.’ Hanoi, December 6th , 2014 Signed.
i LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com ABSTRACT Nowadays, statistical machine translation is derived diverse interest of researchers thanks to its advantages. However, approaches based on statistic constantly confront deficiencies of parallel and specific domain corpora. Generating these corpora re- quires intensive human effort and availability of experts. Unfortunately, only a few popular languages in the world are derived continuous financial support and interest of researchers for development of machine translation systems.
For most remaining languages, there is very small interest of funding available. Therefore it becomes an immense obstacle to apply approaches based on statistic for such languages. The purpose of this thesis is to propose a method for utilizing unannotated corpora to address this impediment. Publications: ? Hien Vu Huy, Phuong-Thai Nguyen, Tung-Lam Nguyen and M.
Bootstrapping Phrase- based Statistical Machine Translation via WSD Integration. In Proceedings of the Sixth Interna- tional Joint Conference on Natural Language Processing (IJCNLP 2013), pp. ii LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com ACKNOWLEDGEMENTS First and foremost, I would like to express my deepest gratitude to my supervi- sor, Dr. Nguyen Phuong Thai, for his patient guidance and continuous supports throughout the years.
He always appears when I need help, and responds to queries so helpfully and promptly. I would like to give my honest appreciation to my best friends in my home town for whatsoever they did for me. I sincerely acknowledge the Vietnam National University, Hanoi and especially, QG.49 project for sup- porting finance to my master study. Finally, this thesis would not have been possible without the support and love of my parents.
Thank you! iii LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com To my family ♥ iv LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com Table of Contents 1 Introduction 1 2 Literature review 4 2.4 Moses - an Open Statistical Machine Translation System .2 Word Sense Disambiguation. 11 3 Utilizing WSD for SMT 17 3.2 WSD Training Data Generation .2 Using Unlabelled Data .2 A new Algorithm with Sense Distribution Control .3 Using Clustering Context Information .1 Corpora and Tools. 26 v LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com TABLE OF CONTENTS vi 4.1 Extend Labelled Data .2 WSD clustering task .3 The impact of context on WSD .4 Impact of WSD system on SMT translation system. 32 5 Conclusion 35 LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com List of Figures 1.1 Integrating WSD into phrase-based SMT system .1 Integrating WSD into phrase-based SMT system .1 Sense distribution of interest.
24 vii LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com List of Tables 4.1 Statistics for training, testing and developing corpora .2 Statistics for training, testing and developing corpora for using clus- tering context information .3 Statistics for samples and features before extending and after extending 28 4.4 Expansion result with the word interest .5 Example translation of the test for hard .6 Example translation of the test for maturity .7 BLEU scores of phrase-based SMT systems with WSD and BNC- extended WSD .8 Accuracy of WSD system with the clustering feature and without the clustering feature for words hard, good, maturity and grow .9 Example translation of the test for late .10 BLEU scores of phrase-based SMT systems with WSD and WSD with the clustering feature .11 Sense distribution of since. 34 viii LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com List of Abbreviations SMT Statistical Machine Translation EBMT Example-based machine translation WSD Word Sense Disambiguation BLEU Bilingual Evaluation Understudy MERT Minimum Error Rate Training MEM Maximum Entropy Model BNC British National Corpus POS Part of Speech AI Artificial Intelligence ix LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com Chapter 1 Introduction Conventional phrase based systems use local context information from phrase table and language model. Although phrase based SMT achieves a jump in translation quality in comparison with word based SMT, there are still cases in which local context can not capture well the correct meaning of source words. WSD can use features from much larger contexts and those features can overlap each other.
The idea of integrating WSD and SMT rises naturally from this perspective. Previously, (Garcı́a-Varea et al., 2001) directly used context sensitive lexical models for SMT. Their SMT system was a word-based maximum entropy model (MEM). They reported significant decreases in perplexities of training and testing corpora.
Besides, they also used these lexical models for re-ranking n-best lists and achieved slight improvements in translation quality. (Chan et al., 2007) made use of WSD for hierarchical phrase-based translation. WSD training data was generated from bilingual corpus using word alignment in- formation. They used two new WSD features for SMT and proposed an algorithm for scoring synchronous rules.
Phrases which do not exceed a length of two were computed WSD models. Their experiments, carried out using a standard Chinese to English translation task, showed that WSD can improve SMT significantly. Simultaneously with (Chan et al., 2007), (Carpuat and Wu, 2007) used a similar approach to the problem. The main difference was that they focused on conventional phrase-based SMT in (Koehn et al., 2003) and used only one WSD feature for SMT.
The limit of phrase length was the same as the value used by their SMT system. Their experiments led to the same conclusion: WSD can improve SMT. However, approaches based on statistic constantly confront deficiencies of parallel 1 LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com 2 and specific domain corpora. Generating these corpora requires intensive human effort and availability of experts.
Only a few popular languages in the world are derived continuous financial support and interest of researchers for development of machine translation systems. For most remaining languages, there is very small interest of funding available. Therefore it becomes an immense obstacle to apply approaches based on statistic for such languages. Recently, there are several approaches to address this impediment.
(Pham et al., 2005) has shown his study on semi-supervised learning algorithms, namely cotraining in (Blum and Mitchell, 1998), smooth cotraining in (Mihalcea, 2004), spectral graph transduction in (Joachims, 2003) and its cotraining variant for WSD. He has used information of surrounding words and local collocations for Senseval 2 English lexical sample task and English all-words task. His results showed that using unlabelled data can bring significant improvement in WSD accuracy. Similarity, (Niu et al., 2005) has used label propagation algorithm in (Zhu and Ghahramani, 2002) for WSD in which unlabelled data and labelled data were represented by vertices in a connected graph.
Then label information was iteratively propagated from any vertex to nearby vertices through weighted edges. After the propagation process converged, unlabelled data was assigned labels. The result indicated that his method outperformed traditional supervised machine learning methods in WSD for sparse data problem. However, both (Pham et al., 2005) and (Niu et al., 2005) have merely used unlabelled data without observing sense distribution of words.
In addition to improve accuracy of WSD, various approaches focus directly on SMT to face sparse data problem. (Yang and Kirchhoff, 2010) utilized a graph-based unsupervised WSD algorithm to approach a lacking of specific domain data. They represented all word senses in a text as nodes in an undirected graph, applying PageRank algorithm to score edges in their graph for calculating correct senses and attaining a slight improvement in quality of translation. (Ambati et al., 2011) applied multi-strategy methods to active learning for ma- chine translation.
They combined several technique in sentences selection process in order to enhance quality of active learning approach. Results in their experiments indicated that their approach enhanced translation performance significantly when parallel training data was scarce. However, most of these mentioned above approaches focus on target language to improve translation quality. In this thesis, we present our study on this topic and propose a method to improve translation quality via source language.
LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com 3 The rest of this thesis is organized as following. First, we introduce an overview of related backgrounds in Chapter 2. Then by integrating WSD as a model of SMT system as shown in the Figure 1.1, we present how we use WSD for SMT in Chapter 3. We also demonstrate a method to bootstrap WSD models by using unlabelled data and to exploit words clustering information in this chapter.
In Chapter 4, we present our experiments and results. Finally, conclusion is shown in Chapter 5.1: Integrating WSD into phrase-based SMT system LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com Chapter 2 Literature review In this chapter, we introduce an overview of Machine Translation (MT) and Word Sense Disambiguation (WSD) in section 2. In the first section, we introduce the history, approaches in MT and Moses SMT tool-kit. In the remaining section, we introduce an overview of WSD.1 The history Machine Translation is a sub-field of computational linguistics which investigate methods to translate text or speech from one natural language to another by uti- lizing software.
The first idea of using machine to translate text came from Leibniz and Descartes in seventh century when they gave an idea to use a general language which can represent one idea in all different languages by one symbol. However, this proposal was not executed. The first invention for automatic translated application was executed in the mid of 1930s. At this time, Georges Artsruni created an bilingual dictionary for an auto- matic look up.
After that, Pyotr Troyanski continuously developed this invention with many complements including grammar rules based on constructed international auxiliary language (Esperanto). In 1950s, the history of modern machine transla- tion was officially recorded. In 1954, A Georgetown MT research team successfully translated automatically 60 Russian sentences into English. This initial succeed opened a new chance for research of machine translation.
At this time, the authors claimed that all issues of machine translation will be solved in next several years 4 LUAN VAN CHAT LUONG download : add luanvanchat@agmail. Machine Translation 5 in (Hutchins et al. However, real progress was much slower than they ex- pected. In a report in (Pierce and Carroll, 1966) pointed out that after ten-year-long research had failed to fulfill expectations, due to which, funding for MT research was greatly reduced.
In the 1980s, research on MT typically relied on translation through some variety of intermediary linguistic representation involving morpholog- ical, syntactic, and semantic analysis. Until the end of 1980s, when computational power increased and became less expensive, more interest appeared and focused on statistical approach which was impossible in 1960s and 1970s. There was a large surge in a number of novel methods for MT in this time.