VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY NGUYEN HAI LONG JOINT LEARNING FOR LEGAL DOCUMENT RETRIEVAL: LEVERAGING THE RELATIONSHIP BETWEEN RELEVANCY AND AFFIRMATION MASTER THESIS HA NOI - 2024 VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY JOINT LEARNING FOR LEGAL DOCUMENT RETRIEVAL: LEVERAGING THE RELATIONSHIP BETWEEN RELEVANCY AND AFFIRMATION MASTER THESIS Major: Computer Science Supervisor: Assoc. Phan Xuan Hieu HA NOI - 2024 Abstract The information retrieval problem is a typical problem in the field of legal AI. This problem could be utilized in many practical applications, help the legal experts as well as the people who do not have the specialized knowledge about legal domain, to conveniently search related legal articles for a question or a legal issue. Finding related articles is an indispensable step, which forms a basis knowledge before determining the correctness of the query.
Thus, in legal text reasoning, the relevancy need to be considered first, and then the affirmation will be determined. While the relevancy and affirmation are two different concepts, there would exists a connection between them. Performing retrieval and textual entailment sequen- tially and independently may not exploit the most of this mutually supportive relationship. Therefore, this thesis propose a multi-task learning approach for these tasks to improve their performance.
Our empirical findings indicate that supportive relationship truly exists and can contribute to the results of the model. This important insight sheds light on how leveraging relationship between tasks can significantly enhance the effectiveness of the multi-task learning approach for legal text processing. In addition to the proposed multi-task approach, this thesis also extends the re- trieval system by adding a re-ranking phase utilizing LLMs as the final phase. Due to the basic logical reasoning capability of LLMs, the re-ranking phase help the retrieval system to handle complicated queries that do not written in legal language and need logical inference for finding the related legal terms and definitions.
The experiments on the LLMs-support retrieval system produces a significant improve- ment in the retrieval results. 11 Acknowledgements First and foremost, I wish to express my sincere thanks to my supervisor Assoc. Phan Xuan Hieu, for his dedicated involvement and continuous encour- agement. I would also like to show my gratitude to Mrs.
Vuong Thi Hai Yen and Dr. Ha-Thanh Nguyen. Without their valuable guidance and encouragement, my thesis would have never been accomplished. In addition, I would take this opportunity to express gratitude to all the Department faculty members including lecturers, and my colleagues for their help and support.
I also thank parents for unceasing encouragement, support and attention. I place on record, my sense of gratitude to one and all, who directly or indi- rectly, have lent their hand in this venture. iv Declaration I declare that the thesis has been composed by myself and that the work has not be submitted for any other degree or professional qualification. I confirm that the work submitted is my own, except where work which has formed part of jointly-authored publications has been included.
My contribution and those of the other authors to this work have been explicitly indicated below. I confirm that appropriate credit has been given within this thesis where reference has been made to the work of others. The model presented in Chapter 3 and the results presented in Chapter 4 was previously published in the Proceedings of NLLP 2023 as “Joint Learning for Legal Text Retrieval and Textual Entailment: Leveraging the Relationship between Rele- vancy and Affirmation” and accepted for publish at JURISIN 2024 as “Enhancing Legal Document Retrieval: A Multi-Phase Approach with Large Language Models” by myself et al. This study was conceived by all of the authors.
I carried out the main idea(s) and implemented all the model(s) and material(s). I certify that, to the best of my knowledge, my thesis does not infringe upon anyone’ s copyright nor violate any proprietary rights and that any ideas, tech- niques, quotations, or any other material from the work of other people included in my thesis, published or otherwise, are fully acknowledged in accordance with the standard referencing practices. Furthermore, to the extent that I have included copyrighted material, I certify that I have obtained a written permission from the copyright owner(s) to include such material(s) in my thesis. I declare that this thesis has not been submitted for a higher degree to any other University or Institution.
Student Nguyen Hai Long Table of contents Abstract. ce ee ee eee ee iii Acknowledgements.Ặ ee HS eee ne iv Declaration. ee ee ee ee ee Vv Table of contents. ee ee ee ee ee Kia vi ACTONYMS Tdaaiiiiiáẳi.a ee ene ix List of Ífgures.
ee ee te ee ee x List of tables. ee ee ee ene xi 1 Introduction. ce ee ee ee es 1 1.1 Legal AI research motivation. Challenges in legal AI with NLP.1 Legal document retrieval (LDR).2 Legal textual entailment (LPE).
ra 7 2 Related works and backgrounds knowledge .1 Related works about legal information retrieval.2 Natural language processing background.1 TF-IDF algorithm .2 Okapi-BM25 algorithm .4 BERT: Bi-directional encoder representation.5 Mono L5 sequence-to-sequence model for document ranking 17 2.26 GPT large language model .7 Weighted cross-entropy loss function. 18 3 Enhancing LDR with multi-task model and LLMs-supported .1 Legal multi-task problem formation .1 Supportive relationship between relevancy and affirmation .2 Sub-entailment: decomposed entailment relationship .3 Legal relevancy-affirmation problem (LRA) .2 Retrieval system utilizing multi-task model.1 Retrieval system components.2 Overall retrieval system 1 .3 Legal retrieval system with LLMs re-ranking .1 Retrieval system 2’s architecture. LLMs prompting template. 30 4 Evaluation on COLIEE dataset.1 COLIEE dataset introduction.2 Retrieval evaluation metric.000002 eae 35 43 Implementation detail .1 Retrieval system 1: lexical and semantic ranking model.2 Retrieval system 2: additional LLMs re-ranking phase.
LH vo 45 ConclusiOns. ee HH ee ee HN KV ee Kia 47 Ablation study. Ặ Q Q HQ Q HQ HH ee ee ee ee es 48 List of Publications. Ặ Q Q HQ ee HH HH KV ee 50 vii References.
- - c c Q Q et te te ee ee ee eee ee eee ee OL vill Acronyms AI Artificial Intelligence BERT Bi-directional Encoder Representations from Transformers BM25 Best Match 25 COLIEE Competition on Legal Information Extraction and Entailment DLTE Decomposed Legal Textual Entailment GPT Generative Pre-trained Transformer IDF Inverse Document Frequency LDR Legal Document Retrieval LLMs Large Language Models LM Language Model LRA Legal Relevancy-Afirmation LTE Legal Textual Entailment MLP Multi-layer Perceptron NLP Natural Language Processing TF Term—Frequency ix List of figures 2.1 Background knowledge used in this study .2 Components in a Transformer block .3 Masked token prediction pre-training task for BERT model 16 2.4 NÑext-sentence prediction pre-training task for BERT model .1 Observation of two legal reasoning processes .2 Two scenarios can happen when using the decomposed textual en- tailment relationship .3 Multi-task BERI-based model architecture.4 Overall pipeline of retrieval system Ì.9 Overall pipeline of retrieval system 2.6 Visualization of sliding window technique.1 Process of pair sample transformation which generates the training data for semantic-based ranking model .2 Post—processing phase for a single query .3 The statistical analysis of avg. error and avg. error correction of the MultiTask model with the Single Task model. List of tables 4.1 Examples from training set of COLIEE 2023 competition.
The origi- nal queries and document corpus are in Japanese. An English trans- lation version is also provided.2 Statute Law Corpus Statistic .4 Recall score of corresponding top-k .5 F2-measure of all models and other teams [14] evaluated on COLIEE 2022 dataset. LH ng kg kg va 4.6 F2-measure of all models and other teams [11] evaluated on COLIEE 2023 dataset.7 Experiment results on COLIEE 2023.8 Results of participating teams in the COLIEE 2023. Xi Chapter 1: Introduction Recently, legal AT is a rapidly developing field that brings advanced AI technologies to enhance legal expertise.
This chapter outlines the motivation behind utilizing artificial intelligence to improve typical legal tasks. Next, the challenges in applying NLP techniques posed by the legal texts’ characteristics and the definitions of the target problems are also described. By leveraging the supportive relationship between the relevancy and affirmation, as well as the logical inference capability of LLMs, this thesis aims to develop an efficient retrieval system that can assist both legal professionals and the general public in finding relevant legal documents with moderate model size and high accuracy. Legal AI research motivation There are two types of legal systems: civil law and common law, whereas each country in the world uses one of these legal systems or a mix of both systems.
While the common laws come from the case laws which are the result of legal judicial decisions in the past, the civil law system comes from the pre-defined law, which is employed for handling cases in the court, as well as the basis for the state to execute its actions. In countries that utilize the civil law system, understanding legal matters is very essential, which prevents people from acting illegally and creates negative impacts on oneself and society. Thus, legal assistance which helps not only legal experts but also people to find related legal articles with a natural language query, is very necessary. This study focuses on the context of legal article retrieval problem in the civil law system, the goal is to create an accurate and efficient legal retrieval system.
Nowadays, the fast development of the NLP field with the emergence of foundation models such as BERT |6], RoBERTa [16] and GPT-4 |4] presents a significant opportunity for advancement in legal AI research. This area is highly promising and has the potential to create numerous useful applications not only for lawyers and legal professionals but also for the general public. The statute law retrieval problem is one of the typical challenges in the legal AI field, with many practical applications. Retrieving articles related to a specific issue or statement can provide a wealth of information for people to make critical decisions.
The legal textual-entailment problem is also an essential problem in Legal AI, which involves determining whether a query entails the content of related arti- cles. The observation comes from the reality that sometimes a lawyer searches for articles related to a query to substantiate their view on the query’s correctness/in- correctness, which indicates that the textual-entailment property would have a very close relationship with the relevance property. So, integrating information about relevance and textual-entailment to develop a retrieval system is a promis- ing idea. Here is a sample from the Japanese legal bar exam, which contains a query, related articles retrieved from the Japanese statute law, and the entailment relationship: ¢ Query: In cases where an individual rescues another person from getting hit by a car by pushing that person out of the way, causing the person’s luxury kimono to get dirty, the rescuer does not have to compensate damages for the kimono.
¢ Related articles: — Article 698: Jf a manager engages in benevolent intervention in an- other’s business to allow a principal to escape imminent danger to the principal’s person, reputation, or property, the manager is not liable to compensate for damage resulting from this unless the manager has acted in bad faith or with gross negligence. So far, many studies address the legal document retrieval problem using lexical features combined with semantic features [15, 18, 28]. However, this research in- dicates that the entailment relationship implied for the affirmation property is an important factor, that could raise significantly the retrieval efficiency without increasing the model’s size or using other complicated techniques. Another motivation of this thesis comes from the recent breakthroughs of LLMs, which are marked by the appearance of generative language models like GPT-4 [1], Gemini [31], Mistral [12], Llama [32], etc,.
These large language mod- els have an almost perfect natural language fluency capability, the basic logical reasoning in some complicated tasks, and the ability to understand the intention 2 behind the user’s input to perform it correctly.