VIETNAM NATIONAL UNIVERSITY, HANOI INTERNATIONAL SCHOOL GRADUATION PROJECT PROJECT NAME AUGMENTING THE COMPUTATIONAL AND REASONING PROFICIENCIES OF LARGE LANGUAGE MODELS FOR TACKLING VIETNAMESE HIGH SCHOOL MATHEMATICAL PROBLEMS Nguyễn Thảo Vi Hanoi - Year 2024 VIETNAM NATIONAL UNIVERSITY, HANOI INTERNATIONAL SCHOOL GRADUATION PROJECT AUGMENTING THE COMPUTATIONAL AND REASONING PROFICIENCIES OF LARGE LANGUAGE MODELS FOR TACKLING VIETNAMESE HIGH SCHOOL MATHEMATICAL PROBLEMS SUPERVISOR: Dr. Nguyen Doan Dong STUDENT: Nguyen Thao Vi STUDENT ID: 20070011 COHORT: QH2020 SUBJECT CODE: INS401101 MAJOR: Management Information Systems Hanoi - Year 2024 2 ACKNOWLEDGEMENTS I would like to extend my sincere appreciation to my thesis supervisor, Dr. Nguyen Doan Dong, for his unwavering support, invaluable guidance, and genuine enthusiasm throughout this thesis research. His dedicated mentorship has been instrumental in shaping the direction and quality of this work, enabling me to acquire valuable knowledge and develop a profound sense of self-assurance in my chosen field of study.
I am deeply grateful to the Faculty of Applied Science for providing the necessary resources and fostering a conducive learning environment that has facilitated the successful completion of this research. The faculty members, particularly those within the International School, have imparted invaluable knowledge and skills, enriching my academic journey. Furthermore, I would like to express my heartfelt thanks to my family and friends for their steadfast support and encouragement during this challenging endeavor. Their belief in my abilities and constant encouragement has been a constant source of strength and motivation.
Completing a graduation thesis is a demanding undertaking, and I acknowledge that I could not have reached this academic milestone without the companionship, support, and guidance of all those mentioned above. I am sincerely grateful to each and every one of you for being an integral part of this remarkable journey. 3 DECLARATION I, Nguyen Thao Vi, hereby declare that the thesis entitled "Augmenting the Computational and Reasoning Proficiencies of Large Language Models for Tackling Vietnamese High School Mathematical Problems", presented as a partial fulfillment of the requirements for the degree of Management Information Systems at International School, Vietnam National University, Hanoi, is entirely my original work. The ideas, data, and information presented in this thesis have been appropriately sourced and referenced.
The plagiarism rate in this thesis is 16%, as determined through thorough evaluation and verification by my supervising professor, satisfying the requirement set by the Judgement Board of less than 20%,. This thesis reflects my independent thinking and comprehensive understanding of the subject matter, without any falsification, fabrication, or manipulation of data or results. All images, figures, tables, and other materials utilized in this thesis have either been created by myself or have been sourced from the public domain, with proper permissions and acknowledgments from the original creators. I am fully aware that any violation of the aforementioned declarations may lead to consequences in accordance with the academic policies and integrity guidelines of International School, Vietnam National University, Hanoi.
Nguyen Thao Vi 2024 4 List of acronyms Acronym Meaning LLM Large Language Model ICL In-context Learning SR Samples Retrieval CoT Chain-of-thought Q Question A Answer G Grade T Topic SP Sample Problem SA Sample Answer CS Code String CR Code Result CRF Conditional Random Field TF-IDF Term Frequency-Inverse Document Frequency KNN K-Nearest Neighbors MLP Multi-layer Perceptron ACC Accuracy TP True Positives TN True Negatives FP False Positives FN False Negatives API Application Programming Interface 5 List of tables Table no. Table name Table 3-1 Math QA datasets overview Table 3-2 Model Performance Summary on three datasets Table 3-3 Math Topic Grade 10 Classification Report Table 3-4 Math Topic Grade 11 Classification Report Table 3-5 Math Topic Grade 12 Classification Report Table 4-1 Experiment Result Summary List of figures Figure no. Figure name Figure 3-1 Zero-shot GPT-4 Method (Returns incorrect answer) Figure 3-2 The Proposed Method (Returns correct answer) Figure 3-3 Topic Distribution for Grade10_QA Figure 3-4 Topic Distribution for Grade11_QA Figure 3-5 Topic Distribution for Grade12_QA Figure 3-6 Prompt template for code generation Figure 3-7 Code Execution Process via API Figure 5-1 The Architecture of my Math Solver Web App Figure 5-2 Web App Interface for Question Input Figure 5-3 Web App Interface for Answer Output 6 TABLE OF CONTENTS CHAPTER 1 - INTRODUCTION .9 CHAPTER 2 - LITERATURE REVIEW. Overview of large language models.
LLMs performance assessment on math. Challenges of solving Vietnamese high school mathematics with LLM. Previous research on computational problem-solving and reasoning in mathematics. Reasoning ability improvement via retrieval of similar math problems.
Rationale for topic classification. Sample problems dataset preparation. Model selection for topic classification. Similar math problem retrieval.
Implementation for enhanced computation .38 CHAPTER 4 – EXPERIMENT ANALYSIS. Implementation of the study. Limitations and Future directions .56 7 ABSTRACT This research focuses on addressing the challenges of applying large language models (LLMs) to solve high school mathematics problems, particularly in the Vietnamese context. Despite the remarkable advancements in natural language processing enabled by LLMs, they still struggle when it comes to mathematical problem-solving.
The objective of this study is to enhance the computational and reasoning proficiencies of LLMs for tackling high school mathematics problems. The proposed approach involves utilizing topic classification to identify the relevant mathematical topic, searching for similar problems in a corpus, extracting answers, and employing in-context learning for further improvement. Code generation is also incorporated to facilitate computational problem-solving and provide comprehensive explanations of the outcomes. By overcoming the limitations of LLMs in understanding numbers, computational abilities, and reasoning skills, this research aims to enhance the accuracy and effectiveness of LLM-based approaches in supporting students' mathematical studies.
These findings have the potential to contribute to the advancement of educational technology, assisting students in confidently navigating the intricacies of high school mathematics. 8 CHAPTER 1 - INTRODUCTION Large language models have demonstrated remarkable capabilities in text processing and generation, opening exciting avenues for their application in solving mathematical problems [39,68]. Recent studies have explored the potential of LLMs in tackling various mathematical tasks, including solving basic arithmetic and geometric problems, generating proofs for simple theorems, and even translating natural language descriptions into symbolic expressions [11,20,70,72]. However, the performance of LLMs in solving complex mathematical problems, particularly those encountered in higher-level education settings, remains significantly hindered by several factors.
One key challenge is the inherent complexity of mathematics itself. High school mathematics in Vietnam, for instance, encompasses a diverse range of topics, including Arithmetic, Algebra, Trigonometry, Statistics and Probability, Calculus, and Geometry, with varying degrees of difficulty [6]. This diversity poses a significant challenge for LLMs, as they require a deep understanding of each topic and the ability to apply the relevant knowledge to specific problems. Additionally, many Vietnamese high school math problems involve intricate problem-solving steps and require clear explanations relying on knowledge acquired from prior classes or topics, which can be challenging for LLMs to generate accurately and systematically [40].
Furthermore, previous research has highlighted limitations in the applicability of existing LLM approaches to Vietnamese high school mathematical problems. A study compared the performance of several LLMs in solving problems from the Vietnamese National High School Graduation Exam [65]. While the results demonstrated some promise, the overall accuracy and explainability of LLM solutions remained suboptimal. This highlights the need for novel approaches that specifically address the challenges of applying LLMs to the nuanced context of Vietnamese high school mathematics.
9 Therefore, the primary objective of this research is to develop methodologies that enhance the computational and reasoning capabilities of LLMs in effectively solving Vietnamese high school mathematics problems. This entails leveraging topic classification techniques to accurately identify the mathematical domain of a given question, retrieving the most relevant problem from a corpus, extracting the corresponding answer, and employing in-context learning to refine the model's performance. Additionally, the research seeks to incorporate code-generation techniques to facilitate accurate and efficient computational problem-solving, while also providing comprehensive explanations for the obtained outcomes. The motivation for this research stems from recognizing the potential of LLMs to enable personalized, interactive, and dynamic learning experiences for students, while also automating the generation of comprehensive solutions and explanations for complex problems.
By incorporating LLMs into the educational process, intelligent tutoring systems can be developed that adapt to the unique needs of students, providing them with step-by-step guidance and tailored explanations. This personalized approach enhances student engagement and comprehension, while also saving valuable time for educators and elevating the overall quality of instruction. This study will focus on Vietnamese high school mathematics for grades 10, 11, and 12, encompassing the diverse range of topics covered within the Vietnamese high school curriculum. A curated dataset of Vietnamese high school math problems and solutions will be utilized to train and evaluate the proposed system.
This research introduces a significant contribution by combining reasoning enhancement techniques with case retrieval and code-based computational power enhancement. In previous studies, similar approaches were explored, such as Meta- CoT [74] for reasoning enhancement and Mathprompter [21] for computational power enhancement. However, the integration of both methods had not been attempted before in the context of Vietnamese high school mathematics. Moreover, the first-ever solution specifically tailored to Vietnamese high school mathematics 10 was developed as part of this research.
The data necessary for the study was collected, and the proposed methodologies were tested using this data. To train and evaluate the proposed system, a carefully curated dataset of Vietnamese high school math problems and solutions was utilized, serving as a solid foundation for the research. This dataset not only acts as a valuable resource for future studies in the field but also establishes a benchmark for assessing the performance of large language models in the context of Vietnamese high school mathematics. Overall, this research effectively addresses the limitations and challenges faced by large language models in solving complex mathematical problems.
The integration of reasoning enhancement and code-based computational power enhancement represents notable contributions, as does the development of a tailored solution for Vietnamese high school mathematics. The effort invested in data collection and testing demonstrates a dedicated approach to meeting the specific needs of Vietnamese students and educators. 11 CHAPTER 2 - LITERATURE REVIEW 2. Overview of large language models LLMs have emerged as a groundbreaking advancement in the field of artificial intelligence, revolutionizing the way we process and generate human language.
These remarkable models have been meticulously designed to mimic the natural patterns of human communication [62]. This is achieved through the utilization of neural networks, complex systems inspired by the structure and functioning of the human brain. With billions of parameters, these networks within LLMs possess a remarkable capacity to handle vast amounts of information [39]. Central to their success is the transformative Transformer architecture, which enables LLMs to capture contextual relationships and dependencies within text.
By incorporating self-attention mechanisms, the Transformer architecture empowers LLMs to process language efficiently, model complex patterns, and generate coherent and contextually relevant responses [59]. This architecture has played a vital role in advancing LLMs and reshaping language understanding and generation. In recent years, we have witnessed an exciting wave of highly impressive LLMs that have captivated the attention of researchers and practitioners alike, owing to their exceptional capabilities. Standout models in this realm include the likes of GPT-4 [45], GPT-3.5 [67], Bard [34], and LLaMA [58], whose performance across a wide spectrum of natural language processing (NLP) tasks has been nothing short of remarkable.
These tasks encompass a rich variety of language-related challenges, each requiring distinct yet sophisticated abilities from the models. One such task where these LLMs shine is question answering, where they exemplify their capacity to provide accurate and insightful answers to user queries, drawing upon their comprehensive understanding of the given context. In the realm of text summarization, these models exhibit their prowess in condensing lengthy pieces of text into concise and informative summaries, distilling the essence of the content while preserving key details and key points.