Parallel Texts Extraction from the Web by Le Quang Hung Faculty of Information Technology University of Engineering and Technology Vietnam National University, Hanoi Supervised by Dr. Le Anh Cuong A thesis submitted in fulfillment of the requirements for the degree of Master of Information Technology December, 2010 LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com Contents ORIGINALITY STATEMENT i Abstract ii Acknowledgements iii List of Figures vi List of Tables vii 1 Introduction 1 1.1 Parallel corpus and its role .2 Current studies on automatically extracting parallel corpus .3 Objectives of the thesis .1 The general framework .2 Structure-based methods .3 Content-based methods. 15 3 The proposed approach 16 3.1 The proposed model .2 Content-based filtering module .1 The method based on cognation .2 The method based on identifying translation seg- ments .3 Structure analysis module. 28 iv LUAN VAN CHAT LUONG download : add luanvanchat@agmail.
40 5 Conclusion and Future Works 41 5. 42 A LIBSVM tool 43 B Relevant publications 44 Bibliography 45 LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com List of Figures 1.1 An example of English-Vietnamese parallel texts.1 General architecture in building parallel corpus.2 The STRAND architecture [1].3 An example of aligning two documents.4 The workflow of the PTMiner system [2].5 The algorithm of translation pairs finder [3].6 Architecture of the PTI system [4].7 An example of the two links in the text.1 Architecture of the Parallel Text Mining system.2 Architecture of a standard Web crawler.3 An example of a candidate pair.4 Description of the process content-based filtering module.5 An example of two corresponding texts of English and Vietnamese.6 The algorithm measures similarity of cognates between a texts pair (Etext, V text).7 Relationships between bilingual web pages.8 The paragraphs can be denoted from HTML pages based on the tag < p >.9 Identifying translation paragraphs.10 A sample code written in Java to perform translation from English into Vietnamese via Google AJAX API.11 Web documents and the source HTML code for two parallel trans- lated texts.12 An example of the publication date feature is extracted from a HTML page.1 Figure for precision and recall measures.2 The format of training and testing data.3 Performance of identifying translation segments method.4 Comparison of the methods. 39 vi LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com List of Tables 1.1 Europarl parallel corpus: 10 aligned language pairs all of which include English.1 Symbols and descriptions .1 URLs from three sites: BBC, VOA News and VietnamPlus. pages downloaded and No.3 Structure-based method.4 Content-based method.5 Method based on cognation.6 Combining structural features and cognate information.7 Identifying translation at document level.8 Identifying translation at paragraph level.9 Identifying translation at sentence level.10 Overall results of each method (P-Precision, R-Recall, F-FS core).
39 vii LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com Chapter 1 Introduction In this chapter, we first introduce about parallel corpus and its role in NLP ap- plications. Current studies, objectives of the thesis and contributions are then presented. Finally, the thesis’ structure is shortly described.1 Parallel corpus and its role Parallel text Different definitions of the term “parallel text” (also known as bitext) can be found in the literature. As common understanding, a parallel text is a text in one language together with its translation in another language.
Dan Tufis [5] gives a definition: “parallel text is an association between two texts in different languages that represent translations of each other”.1 shows an example of English-Vietnamese parallel texts. Parallel corpus A parallel corpus is a collection of parallel texts. According to [6], the simplest case is where two languages only are involved, one of the corpora is an exact translation of the other (e. However, some parallel corpora exist in several languages.
For instance, Europarl parallel corpus [8] which includes versions in 11 European languages as report in Table 1. In addition, the direction of the translation need not be constant, so that some texts in a parallel 1 LUAN VAN CHAT LUONG download : add luanvanchat@agmail.1: An example of English-Vietnamese parallel texts. corpus may have been translated from language L1 to language L2 and others the other way around. The direction of the translation may not even be known.
The parallel corpora exist in several formats. They can be raw parallel texts or they can be aligned texts. The texts can be aligned in paragraph level, sentence level or even in phrase level and word level. The alignment of the texts is useful for different NLP tasks.
Statistical machine translation [9, 10] uses parallel sentences as the input for the alignment module which produces word translation probabil- ities. Cross language information retrieval [11–13] uses parallel texts for deter- mining corresponding information in both questioning and answering. Extracting semantically equivalent components of the parallel texts as words, phrases, sen- tences are useful for bilingual dictionary construction [14, 15]. The parallel texts are also used for acquisition of lexical translation [16] or word sense disambiguation [17].
For most of the mentioned tasks, the parallel corpora are currently playing a crucial role in NLP applications. LUAN VAN CHAT LUONG download : add luanvanchat@agmail.1: Europarl parallel corpus: 10 aligned language pairs all of which include English. Parallel Corpus (L1 -L2 ) Sentences L1 Words English Words Danish-English 1,684,664 43,692,760 46,282,519 German-English 1,581,107 41,587,670 43,848,958 Greek-English 960,356 - 27,468,389 Spanish-English 1,689,850 48,860,242 46,843,295 Finnish-English 1,646,143 32,355,142 45,136,552 French-English 1,723,705 51,708,806 47,915,991 Italian-English 1,635,140 46,380,851 47,236,441 Dutch-English 1,715,710 47,477,378 47,166,762 Portuguese-English 1,681,991 47,621,552 47,000,805 Swedish-English 1,570,411 38,537,243 42,810,628 1.2 Current studies on automatically extracting parallel corpus Nowadays, along with the development the Internet, the Web is really a huge database containing multi-language documents thus it is useful for bilingual texts processing. For that reason, many studies [1–4, 18–22] are paying their attention in mining parallel corpora from the Web.
Basically, we can classify these studies into three groups: content-based (CB) [3, 4, 22], structure-based (SB) [1, 2, 18], and hybrid (combination of the both methods) [19–21]. The CB approach uses the textual content of the parallel document pairs being evaluated. This approach usually uses lexicon translations getting from a bilingual dictionary to measure the similarity of content of the two texts. When the bilin- gual dictionary is available, documents are translated word by word to the target language.
The translated documents then are used to find the best matching par- allel documents by applying similarity scores functions such as cosine, Jaccard, Dice, etc. However, using bilingual dictionary may face difficulty because a word usually has many its translations. Meanwhile, the SB approach relies on analysis HTML structure of pages. This approach uses the hypothesis that parallel web pages are presented in similar structures.
The similarity of the web pages are estimated based on the structural HTML of them. Note that this approach does not require linguistical knowledge. LUAN VAN CHAT LUONG download : add luanvanchat@agmail. Introduction 4 In addition, this approach is very effective in filtering a big number of unmatched documents, as it is quite fast but accuracy.
Nevertheless, it has drawbacks that requires the presentation of two sites with similar content must be presented in the same. From our observation, many sites use the same template to design the Web, the structure of pages is similar but the content of them is different. For that reason, HTML structure-based approach is not applicable in some cases.3 Objectives of the thesis As we have introduced, the parallel corpus is the valuable resource for different NLP tasks. Unfortunately, the available parallel corpora are not only in relatively small size, but also unbalanced even in the major languages [3].
Some resources are available, such as for English-French, the data are usually restricted to gov- ernment documents (e., the Hansard corpus) or newswire texts. The others are limited availability due to licensing restrictions as [23]. According to [24], there are now some reliable parallel corpora: Hansard Corpus1 , JRC-Acquis Parallel Cor- pus2 , Europarl3 , and COMPARA4. However, these resources only exist for some language pairs.
In Vietnam, the NLP is in early stage. The lack of parallel corpora is more severe. The lack of such kind of resource has been an obstacle in the development of the data-driven NLP technologies. There are a few studies of mining parallel corpora from the Web, one of them is presented in [22] (for English-Vietnamese language pair).
On the other hand, the current studies [1–4, 18–21] while extremely useful, they have a few drawbacks as mentioned in Section 1. So, obtaining a parallel corpus with high quality is still a challenge. That is why it still remains a big motivation for many studies on this work. The objective of this research is extracting parallel texts from bilingual web sites of the English and Vietnamese language pair.
We first propose two new methods of designing content-based features: (1) based on cognation, (2) based on identifying translation segments. Then, we combine content-based features with structural features under a framework of machine learning.edu/natural-language/download/hansard/ 2 http://langtech.it/JRC-Acquis.html 3 http://www.org/europarl/ 4 http://www.pt/COMPARA/ LUAN VAN CHAT LUONG download : add luanvanchat@agmail.4 Contributions In our work, we aim to automatically extracting English-Vietnamese parallel texts. As encouraging by [20] we formulate this problem as classification problem to utilize as much as possible the knowledge from structural information and the similarity of content. The most important contribution of our work is that we proposed two new methods of designing content-based features and combined with structural-based features to extract parallel texts from bilingual web sites.
• The first method based on cognation. It is worth to emphasize that different from previous studies [2, 20], we use cognate information replace of word by word translation. From our observation, when translating a text from one language to another, some special parts will be kept or changed in a little. These parts are usually abbreviation, proper noun, and number.
We also use other content-based features such as the length of tokens, the length of paragraphs, which also do not require any linguistically analysis. It is worth to note that by this approach we do not need any dictionary thus we think it can be apply for other language pairs. • The second method based on identifying translation segments use to match translation paragraphs. That will help us to extract proper translation units in bilingual web pages.
Previous studies usually use lexicon translations getting from a bilingual dictionary to measure the similarity of content of the two texts, such as in [4, 20]. This approach may face difficulty because a word usually has many its translations. Differently, we use the Google translator because by using it we can utilize the advantages of a statistical machine translation. It helps to disambiguating lexical ambiguity, translat- ing phrases, and reordering.5 Thesis’ structure Given below is a brief outline of the topics discussed in next sections of this thesis: Chapter 2 - Related works The studies that have close relations with our work are introduced in this chapter.
LUAN VAN CHAT LUONG download : add luanvanchat@agmail. Introduction 6 Chapter 3 - The proposed approach We show our proposed model, including the general architecture of the model, how structural features and content-based features are designed and estimated. Chapter 4 - Experiment This chapter evaluates the goodness and effectiveness of our proposed method for extracting parallel texts from the Web. The performance of our proposed and baseline are presented in here.
Chapter 5 - Conclusion and Future works Final conclusions about our work as a whole and the evaluation of the results in particular are presented, followed by suggestions of possible future work that could be done. Finally, references introduce researches that are closely related to our work. LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com Chapter 2 Related works In this chapter, we outline the general framework in building parallel corpus. Then, we review the studies that have close relations with our work.1 The general framework Figure 2.1: General architecture in building parallel corpus.
7 LUAN VAN CHAT LUONG download : add luanvanchat@agmail.