VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY CAN DUY CAT ADVANCED DEEP LEARNING MODELS AND APPLICATIONS IN SEMANTIC RELATION EXTRACTION MASTER THESIS Major: Computer Science HA NOI - 2019 LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY Can Duy Cat ADVANCED DEEP LEARNING MODELS AND APPLICATIONS IN SEMANTIC RELATION EXTRACTION MASTER THESIS Major: Computer Science Supervisor: Assoc. Ha Quang Thuy Assoc. Chng Eng Siong HA NOI - 2019 LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com Abstract Relation Extraction (RE) is one of the most fundamental task of Natural Language Pro- cessing (NLP) and Information Extraction (IE). To extract the relationship between two entities in a sentence, two common approaches are (1) using their shortest dependency path (SDP) and (2) using an attention model to capture a context-based representation of the sentence.
Each approach suffers from its own disadvantage of either missing or redundant information. In this work, we propose a novel model that combines the ad- vantages of these two approaches. This is based on the basic information in the SDP enhanced with information selected by several attention mechanisms with kernel filters, namely RbSP (Richer-but-Smarter SDP). To exploit the representation behind the RbSP structure effectively, we develop a combined Deep Neural Network (DNN) with a Long Short-Term Memory (LSTM) network on word sequences and a Convolutional Neural Network (CNN) on RbSP.
Furthermore, experiments on the task of RE proved that data representation is one of the most influential factors to the model’s performance but still has many limitations. We propose (i) a compositional embedding that combines several dominant linguistic as well as architectural features and (ii) dependency tree normalization techniques for generating rich representations for both words and dependency relations in the SDP. Experimental results on both general data (SemEval-2010 Task 8) and biomedical data (BioCreative V Track 3 CDR) demonstrate the out-performance of our proposed model over all compared models. Keywords: Relation Extraction, Shortest Dependency Path, Convolutional Neural Net- work, Long Short-Term Memory, Attention Mechanism.
iii LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com Acknowledgements I would first like to thank my thesis supervisor Assoc. Ha Quang Thuy of the Data Science and Knowledge Technology Laboratory at University of Engineering and Technology. He consistently allowed this paper to be my own work, but steered me in the right the direction whenever he thought I needed it. I also want to acknowledge my co-supervisor Assoc.Prof Chng Eng Siong from Nanyang Technological University, Singapore for offering me the internship opportuni- ties at NTU, Singapore and leading me working on diverse exciting projects.
Furthermore, I am very grateful to my external advisor MSc. Le Hoang Quynh, for insightful comments both in my work and in this thesis, for her support, and for many motivating discussions. In addition, I have been very privileged to get to know and to collaborate with many other great collaborators. I would like to thank BSc.
Nguyen Minh Trang and BSc. Nguyen Duc Canh for inspiring discussion, and for all the fun we have had over the last two years. I thank to MSc. Ho Thi Nga and MSc.
Vu Thi Ly for continuous support during the time in Singapore. Finally, I must express my very profound gratitude to my family for providing me with unfailing support and continuous encouragement throughout my years of study and through the process of researching and writing this thesis. This accomplishment would not have been possible without them. iv LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com Declaration I declare that the thesis has been composed by myself and that the work has not be submitted for any other degree or professional qualification.
I confirm that the work submitted is my own, except where work which has formed part of jointly-authored publications has been included. My contribution and those of the other authors to this work have been explicitly indicated below. I confirm that appropriate credit has been given within this thesis where reference has been made to the work of others. The model presented in Chapter 3 and the results presented in Chapter 4 was pre- viously published in the Proceedings of ACIIDS 2019 as “Improving Semantic Relation Extraction System with Compositional Dependency Unit on Enriched Shortest Depen- dency Path” and NAACL-HTL 2019 as “A Richer-but-Smarter Shortest Dependency Path with Attentive Augmentation for Relation Extraction” by myself et al.
This study was conceived by all of the authors. I carried out the main idea(s) and implemented all the model(s) and material(s). I certify that, to the best of my knowledge, my thesis does not infringe upon any- one’s copyright nor violate any proprietary rights and that any ideas, techniques, quota- tions, or any other material from the work of other people included in my thesis, pub- lished or otherwise, are fully acknowledged in accordance with the standard referencing practices. Furthermore, to the extent that I have included copyrighted material, I certify that I have obtained a written permission from the copyright owner(s) to include such material(s) in my thesis and have fully authorship to improve these materials.
Master student Can Duy Cat v LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com Table of Contents Abstract. v Table of Contents. ix List of Figures. xi List of Tables .3 Difficulties and Challenges .5 Contributions and Structure of the Thesis .1 Rule-Based Approaches .1 Feature-Based Machine Learning .2 Deep Learning Methods .4 Distant and Semi-Supervised Methods.
18 vi LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com 3 Materials and Methods .2 Convolutional Neural Network .3 Long Short-Term Memory .2 Overview of Proposed System .3 Richer-but-Smarter Shortest Dependency Path .1 Dependency Tree and Dependency Tree Normalization .2 Shortest Dependency Path and Dependency Unit .3 Richer-but-Smarter Shortest Dependency Path .4 Multi-layer Attention with Kernel Filters .2 Multi-layer Attention .5 Deep Learning Model for Relation Classification .2 CNN on Shortest Dependency Path .3 Training objective and Learning method .4 Model Improvement Techniques. 41 4 Experiments and Results .1 Implementation and Configurations .2 Training and Testing Environment .2 Datasets and Evaluation methods .2 Metrics and Evaluation .3 Performance of Proposed model .2 System performance on General domain .3 System performance on Biomedical data .4 Contribution of each Proposed Component. 56 vii LUAN VAN CHAT LUONG download : add luanvanchat@agmail. 60 List of Publications.
62 viii LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com Acronyms Adam Adaptive Moment Estimation ANN Artificial Neural Network BiLSTM Bidirectional Long Short-Term Memory CBOW Continuous Bag-Of-Words CDR Chemical Disease Relation CID Chemical-Induced Disease CNN Convolutional Neural Network DNN Deep Neural Network DU Dependency Unit GD Gradient Descent IE Information Extraction LSTM Long Short-Term Memory MLP Multilayer Perceptron NE Named Entity NER Named Entity Recognition NLP Natural Language Processing POS Part-Of-Speech ix LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com RbSP Richer-but-Smarter Shortest Dependency Path RC Relation Classification RE Relation Extraction ReLU Rectified Linear Unit RNN Recurrent Neural Network SDP Shortest Dependency Path SVM Suport Vector Machine x LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com List of Figures 1.1 A typical pipeline of Relation Extraction system.2 Two examples from SemEval 2010 Task 8 dataset.3 Example from SemEval 2017 ScienceIE dataset.4 Examples of (a) cross-sentence relation and (b) intra-sentence relation.5 Examples of relations with specific and unspecific location.6 Examples of directed and undirected relation from Phenebank corpus.1 Sentence modeling using Convolutional Neural Network.2 Convolutional approach to character-level feature extraction.3 Traditional Recurrent Neural Network.4 Architecture of a Long Short-Term Memory unit.5 The overview of end-to-end Relation Classification system.6 An example of dependency tree generated by spaCy.7 Example of normalized dependency tree.8 Dependency units on the SDP.9 Examples of SDPs and attached child nodes.10 The multi-layer attention architecture to extract the augmented informa- tion.11 The architecture of RbSP model for relation classification.1 Contribution of each compositional embeddings component.2 Comparing the contribution of augmented information by removing these components from the model .3 Comparing the effects of using RbSP in two aspects, (i) RbSP improved performance and (ii) RbSP yielded some additional wrong results. 58 xi LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com List of Tables 4.1 Configurations and parameters of proposed model.2 Statistics of SemEval-2010 Task 8 dataset.3 Summary of the BioCreative V CDR dataset .4 The comparison of our model with other comparative models on SemEval 2010 Task 8 dataset.5 The comparison of our model with other comparative models on BioCre- ative V CDR dataset .6 The examples of error from RbSP and Baseline models. 59 xii LUAN VAN CHAT LUONG download : add luanvanchat@agmail.com Chapter 1 Introduction 1.1 Motivation With the advent of the Internet, we are stepping in to a new era, the era of information and technology where the growth and development of each individual, organization, and society is relied on the main strategic resource - information. There exists a large amount of unstructured digital data that are created and maintained within an enterprise or across the Web, including news articles, blogs, papers, research publications, emails, reports, governmental documents, etc.
Lot of important information is hidden within these doc- uments that we need to extract to make them more accessible for further processing. Many tasks of Natural Language Processing (NLP) would benefit from extracted information in large text corpora, such as Question Answering, Textual Entailment, Text Understanding, etc. For example, getting a paperwork procedure from a large collection of administrative documents is a complicated problem; it is far easier to get it from a structural database such as that shown above. Similarly, searching for the side effects of a chemical in the bio-medical literature will be much easier if these relations have been extracted from biomedical text.
We, therefore, have urge to turn unstructured text into structured by annotating semantic information. Normally, we are interested in relations between entities, such as person, organization, and location. However, it is impossible for human annotation because of sheer volume and heterogeneity of data. Instead, we would like to have a Relation Extraction (RE) system that annotate all data with the structure of our interest.
In this thesis, we will focus on the task of recognizing relations between entities in unstructured text. 1 LUAN VAN CHAT LUONG download : add luanvanchat@agmail.2 Problem Statement Relation Extraction task includes of detecting and classifying relationship between enti- ties within a set of artifacts, typically from text or XML documents.1 shows an overview of a typical pipeline for RE system. Here we have to sub-tasks: Named Entity Recognition (NER) task and Relation Classification (RC) task. Named Relation Unstructured Entity Classification Knowledge literature Recognition Figure 1.1: A typical pipeline of Relation Extraction system.
A Named Entity (NE) is a specific real-world object that is often represented by a word or phrase. It can be abstract or have a physical existence such as a person, a loca- tion, a organization, a product, a brand name, etc. For example, “Hanoi” and “Vietnam” are two named entities, and they are specific mentions in the following sentence: “Hanoi city is the capital of Vietnam”. Named entities can simply be viewed as entity instances (e., Hanoi is an instance of a city).
A named entity mention in a particular sentence can be using the name itself (Hanoi), nominal (capital of Vietnam), or pronominal (it). Named Entity Recognition is the task of seeking to locate and classify named entity mentions in unstructured text into pre-defined categories. A relation usually denotes a well-defined (having a specific meaning) relationship between two or more NEs. It can be defined as a labeled tuple R(e1 , e2 , ., en ) where the ei are entities in a predefined relation R within document D.
Most relation extrac- tion systems focus on extracting binary relations. Some examples of relations are the relation capital-of between a CITY and a COUNTRY, the relation author-of be- tween a PERSON and a BOOK, the relation side-effect-of between DISEASEs and a CHEMICAL, etc. It is also possible be the n-ary relation as well. For example, the relation diagnose between a DOCTOR, a PATIENT and a DISEASE.