HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY MASTER THESIS Representation learning for Knowledge Graph using Deep Learning methods TONG VAN VINH Vinh.yn School of Information and Communication Technology Supervisor: Assoc. Huynh Quyct Thang Supervisor’s signature Institution: = School of Information and Communication Technology January 12, 2022 Graduation Thesis Assignment. Name: Tong Van Vinh Phone: +84354095052 Vinh.vn; vinhbachkhoait@ gmail.com 20BKHDL-E ftiliation: Hanoi Iniversity af Science and Technology Tong Van Vinh - hereby warrants (hal the work and presentation in this thesis por- formed by myself under the supervision of Assoc. Prof, Huynh Quyet Thang.
All the resulls presented in this thesis are truthful and are nol copied frum any other works. All roferences in this thesis including images, tables, figures, and quotes are clearly and fully documentedin the bibliography. Twill take full responsibility for oven ane copy that violates school regulations. Student Signature and Name Acknowledgement I would like to acknowledge and give my warmest thanks to my supervisor, As- soc.
Huynh Quyet Thang inspired me a lot in my research career path. I also thank Mr. Iluynh Thanh Trung, Dr. Nguyen Quoc Viet Hung, and Dr.
Nguyen Thanh Tam for supporting me in giving birth to my brainchild and challenging myself by submitting it to the top-tier conferences. I would also like to thank my committee members for letting my defense be an enjoyable moment and for your thoughtful comments and suggestion. 1 would also like to give a special thanks to my girlfriend Thu Hue and my fam- ily as a whole for their mental support during my thesis writing process. There is nothing to touch my love to you.
Moreover, in the absence of my friends, ‘Tien Thanh, Trong Tuan, Hong Ngoc, Hieu ‘Tran, Minh Tam, Quang Huy, Quang Thang, Ngo The Huan, | could hardly melt away all the tension from my work. ‘Thanks for always accompanying me through ups and downs. Finally, this work-was funded by Vingroup and supported by Vingroup Innovation Foundation (VINIF} under project code VINIF. 1 enormously appreciate all the financial support from Vingroup, allowing me to stay focused on my research without worrying about my financial burden.
Abstract Knowledge graphs (K Gs) have received significant attention in recent years. Gain- ing more profound insight into the structure of knowledge graphs allows us to tackle many challenging tasks, such as knowledge graph alignment, knowledge graph completion, and question answering. Recently, deep leaming methods us- ing the representation of knowledge graph entities (nodes) and relations (edges) in vector space have gained traction from the research community because of their flexibility and prospective performance. The best way to evaluate how good a representation learning method is to use that representation to solve real-world tasks, In terms of knowledge graphs, we can rank methods by their performance on tasks such as knowledge graph completion (KGC) or knowledge graph align- ment (KGA).
However, many rescareh challenges still exist, such as cnhancing the accuracy or simultaneously solving multiple tasks. With such motivation, in the scope of our Master work, we address the three groups of crucial challenges in knowledge graph representation, namely (i) chal- lenges in enhancing KGC performance, (ii) challenges in enhancing KGA per- formance, and (iii) challenges in enhancing both KGC and KGA simultaneously. For the first class of challenges, we develop a model named. NoGE which takes take advantage of not only the power of Graph Neural Networks (GNNs) but also the expressive power of qualerniun vector space and (he co-occurrence slalistic: of elements in KGs to achieve SOTA performance on the KGC task.
Moving to the second challonge group, we proposc EMGCN, a spocial GNN architecture designed to exploil different lypes of information to betler the final alignment re- sults. Finally, we propose IKAMI, the first mullitusk-learaing moucl, to solve the two Lasks simullancously. Our proposed (echniques improve upon the stale-of- the-art for diffcrant tasks and thus cover an extensive range of applications. Student Signature and Name 2.3 Incomplete knowledge graph alignment CHAPTER 3.
ENHANCING KNOWLEDGE GRAPH COMPLETION PERFORMANCE.2, Dual quatornion background B. ENHANCING KNOWLEDGE GRAPII ALIGNMENT PERFORMANCE 4.2 Overview of the Proposed Approach 4.2 The entity alignment framework.3 Relation-aware Multi-order Embedding .1 GCN-based embedding model 4.1 Single-order alignment matrices 4.2 Multi-order alignment matrix 4.4 Puting It All Together.2 End-to-end comparison.6 Robustness to constraint violations CHAPTER 5. MULTITASK LEARNING FOR KNOWLEDGE GRAPH COMPLETION AND KNOWLEDGE GRAPH ALIGNMENT.2 Incomplete Knowledge Graph Alignment 3.2 Outline of the Alignment Process .3 Feature channel models.2 Transitivily-based channel .3 Proximity-based channel.4 The complete alignment process .2 Missing triples recovery. LIST OF FIGURES An illnstration of knowledpe graph.
12 An example ofknowiedae graph completion. 13 An example of knowledge graph entity alignment. 14 Aligning incomplete KGs across domains 15 Encoder Decoder architecture for GNN based models. 21 CNN and GCN comparison[37].
Anillustration of cur proposed NoGE. 41 Overview of EMGCN framework. ee ee ee 43 Different supervision percentage 44 #GCN-layers. 46 Robustness to violations of entity consisteney.
47 Robusiness to violations of relation consistency 51 Framcwork Ovtrvicw.2 Running time (in log seale) ơn different datasetls. 63 53 Saving of labelling effort for entity alignment on D-W-V1 test set 65 54 Robusiness of graph alignment models against nuisc on EN-DE- V2 lest scl eee 5. The model pays less attention to noisy relaions.6 KGC performance comparison between TransK and [KAMI dur- ing training.4 Puting It All Together.2 End-to-end comparison.6 Robustness to constraint violations CHAPTER 5. MULTITASK LEARNING FOR KNOWLEDGE GRAPH COMPLETION AND KNOWLEDGE GRAPH ALIGNMENT.2 Incomplete Knowledge Graph Alignment 3.2 Outline of the Alignment Process .3 Feature channel models.2 Transitivily-based channel .3 Proximity-based channel.4 The complete alignment process .2 Missing triples recovery.
LIST OF TABLES 3.3 41 Statistics of real-world datasetS. 42 End to cnd comparison. eee 44 Different weighting schemes of GCN layes. 45 Kffects of similarity matrix coefficiens.
Summary oŸ ntation u8ed. Dalaset statistics for KG alignment End-to-end KG alignment performance (bold: winner, underline: firstrunner-up). ee ee eee Ablation study 2. eee Knowledge Graph Completion performance 2.
Correct aligned relations in EN: >FR KGs. TABLE OF CONTENTS: CHAPTER 1.2 Knowledge graph completion and knowledge graph alignment.1 Knowledge graph completion.2 Knowledge graph alignment.3 The relation between completion and alignment .1 Handle knowledge graph completion challenges .2 Handle knowledge graph alignment challenges 6 1.3 Handle the challenges of solving the two task simultaneously .5 Contributions and Thesis Outline 8 1.1 Graph Convolutional Networks {GCNs).2 Knowledge Graph Completion background.1 Incomplete knowledge graphs 2.2 Knowledge graph completion models .3 Knowledge Craph Alignmenl background.1 Previous approaches LIST OF TABLES 3.3 41 Statistics of real-world datasetS. 42 End to cnd comparison. eee 44 Different weighting schemes of GCN layes.
45 Kffects of similarity matrix coefficiens. Summary oŸ ntation u8ed. Dalaset statistics for KG alignment End-to-end KG alignment performance (bold: winner, underline: firstrunner-up). ee ee eee Ablation study 2.
eee Knowledge Graph Completion performance 2. Correct aligned relations in EN: >FR KGs. TABLE OF CONTENTS: CHAPTER 1.2 Knowledge graph completion and knowledge graph alignment.1 Knowledge graph completion.2 Knowledge graph alignment.3 The relation between completion and alignment .1 Handle knowledge graph completion challenges .2 Handle knowledge graph alignment challenges 6 1.3 Handle the challenges of solving the two task simultaneously .5 Contributions and Thesis Outline 8 1.1 Graph Convolutional Networks {GCNs).2 Knowledge Graph Completion background.1 Incomplete knowledge graphs 2.2 Knowledge graph completion models .3 Knowledge Craph Alignmenl background.3 Link-augmented taining process.2 End-tu-cnd comparison.3 Robustness to KGs incompleteness.4 Saving of labelling effort .3 Link-augmented taining process.2 End-tu-cnd comparison.3 Robustness to KGs incompleteness.4 Saving of labelling effort. CONCLUSION LIST OF TABLES 3.3 41 Statistics of real-world datasetS.
42 End to cnd comparison. eee 44 Different weighting schemes of GCN layes. 45 Kffects of similarity matrix coefficiens. Summary oŸ ntation u8ed.
Dalaset statistics for KG alignment End-to-end KG alignment performance (bold: winner, underline: firstrunner-up). ee ee eee Ablation study 2. eee Knowledge Graph Completion performance 2. Correct aligned relations in EN: >FR KGs.
LIST OF FIGURES An illnstration of knowledpe graph. 12 An example ofknowiedae graph completion. 13 An example of knowledge graph entity alignment. 14 Aligning incomplete KGs across domains 15 Encoder Decoder architecture for GNN based models.
21 CNN and GCN comparison[37]. Anillustration of cur proposed NoGE. 41 Overview of EMGCN framework. ee ee ee 43 Different supervision percentage 44 #GCN-layers.
46 Robustness to violations of entity consisteney. 47 Robusiness to violations of relation consistency 51 Framcwork Ovtrvicw.2 Running time (in log seale) ơn different datasetls. 63 53 Saving of labelling effort for entity alignment on D-W-V1 test set 65 54 Robusiness of graph alignment models against nuisc on EN-DE- V2 lest scl eee 5. The model pays less attention to noisy relaions.6 KGC performance comparison between TransK and [KAMI dur- ing training.3 Incomplete knowledge graph alignment CHAPTER 3.
ENHANCING KNOWLEDGE GRAPH COMPLETION PERFORMANCE.2, Dual quatornion background B. ENHANCING KNOWLEDGE GRAPII ALIGNMENT PERFORMANCE 4.2 Overview of the Proposed Approach 4.2 The entity alignment framework.3 Relation-aware Multi-order Embedding .1 GCN-based embedding model 4.1 Single-order alignment matrices 4.2 Multi-order alignment matrix 4.4 Puting It All Together.2 End-to-end comparison.6 Robustness to constraint violations CHAPTER 5. MULTITASK LEARNING FOR KNOWLEDGE GRAPH COMPLETION AND KNOWLEDGE GRAPH ALIGNMENT.2 Incomplete Knowledge Graph Alignment 3.2 Outline of the Alignment Process .3 Feature channel models.2 Transitivily-based channel .3 Proximity-based channel.4 The complete alignment process .2 Missing triples recovery. TABLE OF CONTENTS: CHAPTER 1.2 Knowledge graph completion and knowledge graph alignment.1 Knowledge graph completion.2 Knowledge graph alignment.3 The relation between completion and alignment .1 Handle knowledge graph completion challenges .2 Handle knowledge graph alignment challenges 6 1.3 Handle the challenges of solving the two task simultaneously .5 Contributions and Thesis Outline 8 1.1 Graph Convolutional Networks {GCNs).2 Knowledge Graph Completion background.1 Incomplete knowledge graphs 2.2 Knowledge graph completion models .3 Knowledge Craph Alignmenl background.1 Previous approaches LIST OF FIGURES An illnstration of knowledpe graph.
12 An example ofknowiedae graph completion. 13 An example of knowledge graph entity alignment. 14 Aligning incomplete KGs across domains 15 Encoder Decoder architecture for GNN based models. 21 CNN and GCN comparison[37].
Anillustration of cur proposed NoGE. 41 Overview of EMGCN framework. ee ee ee 43 Different supervision percentage 44 #GCN-layers. 46 Robustness to violations of entity consisteney.
47 Robusiness to violations of relation consistency 51 Framcwork Ovtrvicw.2 Running time (in log seale) ơn different datasetls. 63 53 Saving of labelling effort for entity alignment on D-W-V1 test set 65 54 Robusiness of graph alignment models against nuisc on EN-DE- V2 lest scl eee 5. The model pays less attention to noisy relaions.6 KGC performance comparison between TransK and [KAMI dur- ing training. ee ee LIST OF TABLES 3.3 41 Statistics of real-world datasetS.
42 End to cnd comparison. eee 44 Different weighting schemes of GCN layes. 45 Kffects of similarity matrix coefficiens. Summary oŸ ntation u8ed.
Dalaset statistics for KG alignment End-to-end KG alignment performance (bold: winner, underline: firstrunner-up). ee ee eee Ablation study 2. eee Knowledge Graph Completion performance 2. Correct aligned relations in EN: >FR KGs.
LIST OF TABLES 3.3 41 Statistics of real-world datasetS. 42 End to cnd comparison. eee 44 Different weighting schemes of GCN layes. 45 Kffects of similarity matrix coefficiens.
Summary oŸ ntation u8ed. Dalaset statistics for KG alignment End-to-end KG alignment performance (bold: winner, underline: firstrunner-up). ee ee eee Ablation study 2. eee Knowledge Graph Completion performance 2.
Correct aligned relations in EN: >FR KGs .3 Link-augmented taining process.2 End-tu-cnd comparison.3 Robustness to KGs incompleteness.4 Saving of labelling effort. CONCLUSION LIST OF FIGURES An illnstration of knowledpe graph. 12 An example ofknowiedae graph completion. 13 An example of knowledge graph entity alignment.
14 Aligning incomplete KGs across domains 15 Encoder Decoder architecture for GNN based models. 21 CNN and GCN comparison[37]. Anillustration of cur proposed NoGE. 41 Overview of EMGCN framework.
ee ee ee 43 Different supervision percentage 44 #GCN-layers. 46 Robustness to violations of entity consisteney. 47 Robusiness to violations of relation consistency 51 Framcwork Ovtrvicw.2 Running time (in log seale) ơn different datasetls.