HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY THESIS Application of deep learning and text embedding methods for self-admitted technical debt detection TRAN THI DINH dinh.vn Thesis advisor : Dr. Bui Thi Mai Anh Signature of advisor Department : Department of Software Engineering Institute : School of Information and Communication Technology Hanoi, 10-2023 THESIS ASSIGNMENT 1. Student’s information : Name : Tran Thi Dinh. Phone : 0971236392 Email: dinh.vn Class : Data Science (Elitech) Affiliation : Hanoi University of Science and Technology.
Thesis title : Application of deep learning and text embedding methods for self- admitted technical debt detection 3. Declarations/Disclosures : I herewith formally declare that I — Tran Thi Dinh — have performed the work and presentation in this thesis independently under supervisions of Dr. Bui Thi Mai Anh. All of the results are genuine and are not copied from any other sources.
Every reference materials are clearly listed in the bibliography. I will accept full responsibility for even one copy that violates school regulations. Hanoi, date month year 2023 Author Tran Thi Dinh 4. Attestation of thesis advisor:.
Hanoi, date month year 2023 Thesis Advisor Dr. Bui Thi Mai Anh Acknowledgments I would like to take this moment to express my deep and heartfelt gratitude to the individuals whose unwavering support, invaluable guidance, and unwavering assistance have been the cornerstone of my successful journey in completing this thesis. Foremost, I extend my sincerest thanks to Dr.Bui Thi Mai Anh and Dr.Nguyen Thanh Phuong, whose mentorship has been a beacon of wisdom and expertise. Their continuous support and mentorship have not only illuminated the path of this research but have also significantly contributed to its depth and quality.
The insightful feedback and constructive critique they have generously provided have not only steered my work in the right direction but have also fostered my academic growth. Furthermore, beyond these formal acknowledgments, I wish to express my gratitude to my friends and family, whose unwavering support, encouragement, and invaluable in- sights have accompanied me throughout this thesis journey. Your camaraderie and moti- vation have been a constant source of inspiration, reminding me of the strength derived from community and shared aspirations. In conclusion, I humbly acknowledge that this thesis would have remained a dream without the unwavering support and collaboration of these remarkable individuals.
Their collective contributions have elevated the quality and depth of this research, and I am profoundly thankful for their unwavering dedication to my academic voyage. i Thesis advisor : Dr. Bui Thi Mai Anh Tran Thi Dinh ABSTRACT Software teams typically turn to sub-optimal solutions that deviate from the best soft- ware development principles in order to strike a balance between short-term efficiency and long-term stability. Such solutions might lead to maintenance issues, so called Technical Debt (TD), which need be paid later on.
Previous studies have leveraged text-mining techniques for automated TD detection in source code comments (a.a Self Admitted Technical Debt–SATD), primarily focusing on object-oriented languages like Java. However, SATD detection becomes challenging in scripting languages such as R, which employ dynamic programming paradigms and have highly compact and algorithm-aligned comments. In this thesis, we introduce DebtSniffer as a practical approach for detecting SATD in both R packages and Java source codes. We utilize a code-embedding technique, i., pre-trained BERT models, to retain the rich semantic information embedded in R source code comments and Java source codes.
Subsequently, we apply graph convolutional net- works to establish connections between scattered comment sentences and learn represen- tations for both labeled training data and unlabeled test data by propagating label impact through the graph convolution. To assess the performance of DebtSniffer, we conducted experiments over 4,961 R comments from 503 open source projects which were typically categorized into 12 TD classes and four Java sources: source code comments, commit messages, pull requests, and issue tracking systems. The experimental results show that DebtSniffer accurately identifies SATD, outper- forming the current state-of-the-art approaches based on traditional word embedding tech- niques. Keywords: Self-Admitted Technical Debt, Pretrained BERT model, Graph Convolu- tional Network, Software Engineering ii Contents Abstract ii List of Figures v List of Tables vi List of Acronyms vii 1 Introduction 1 1.4 Organization of Thesis .1 General techniques for SATD detection .2 SATD detection in the R language .3 SATD detection in the Java language .1 Convolutional Neural Networks (CNN) .2 Graph Convolutional Networks (GCN) .3 Text embedding models.
24 iii Thesis advisor : Dr. Bui Thi Mai Anh Tran Thi Dinh 3.4 Transfer Learning for Code Tasks .2 Pretrained LM with CNN Model .3 Pretrained LM with GCN Model (DebtSniffer) .1 Dataset and baselines .2 Results and Discussion .1 Effectiveness of DebtSniffer on R dataset .2 Effectiveness of DebtSniffer on Java datasets .4 Threats to validity. 44 6 Conclusions and Future works 45 6. 46 References 51 Appendices 52 A Term Frequency - Inverse Document Frequency 52 B Pointwise Mutual Information 54 iv List of Figures 1.1 An example of Self-Admitted Technical Debt of the Defects type in R .2 Real-world scenarios where SATD has caused substantial problems .3 Example - SATD Type: Algorithm in Java .1 A basic convolutional neural network (CNN) architecture.1 The graph convolutional neural network.2 The graph convolutional neural network for text data.1 The SATD preprocessing workflow.1 The architecture of Pretrained LM with CNN model.
Pretrained Code- BERT is used as an example.2 The input representation of BERT model.1 The architecture of Pretrained LM with GCN model.2 Schematic of GCN model.1 Comment length distribution in R dataset.2 SATD length distribution in Java dataset.1 Influence of the number of GC layers in SATD detection for R packages. The baseline is equal to zero number of GC layers.2 Influence of the number of GC layers in SATD detection for Code com- ment source in Java. The baseline is equal to zero number of GC layers. 43 v List of Tables 1.1 Taxonomy TD definitions, based on Codabux et al [5] .2 Types of SATD with Examples .1 Statistics of the dataset.2 Number of different types of SATD .1 Comparison Results in R dataset (%) .2 Code comment dataset.3 Pull request dataset.4 Commit message dataset.6 Influence of the number of GC layers on average F1-score of R language dataset and Java Code comment data.
42 vi List of Acronyms CNN Convolutional Neural Network. GCN Graph Convolutional Network. LM Language Models. NLP Natural Language Processing.
OOP Object Oriented Programming. PMI Pointwise Mutual Information. PPMI Positive Pointwise Mutual Information. RNN Recurrent Neural Networks.
SATD Self-Admitted Technical Debt. SOTA State-Of-The-Art. SVM Support Vector Machines. TD Technical Debt.
TF-IDF Term Frequency – Inverse Document Frequency. vii Chapter 1 Introduction SATD refers to code artifacts within a software project where developers explicitly ac- knowledge the presence of suboptimal or problematic code but do not immediately ad- dress it. SATD can manifest as code comments, such as ”// TODO” or ”// FIXME,” and typically reflects issues related to design flaws, code smells, or deferred maintenance. The identification of SATD instances is crucial for several reasons.
It allows development teams to prioritize technical debt repayment, maintain code quality, and reduce the risk of project delays and increased maintenance costs.1 Problem Statement In the dynamic realm of software development, a common challenge that developers fre- quently encounter is the inexorable march of time. The pressures of deadlines and project constraints often force them to make expedient decisions and adopt shortcuts to expedite the development process. However, while these shortcuts may offer immediate relief, they can potentially sow the seeds of long-term consequences [30]. These expedient measures, taken in the heat of project development, can inadvertently give rise to a myriad of issues.
One of the foremost concerns is the inadvertent creation of low-quality code. These hasty coding practices, often driven by the urgency of project timelines, may result in code that lacks the robustness, efficiency, and maintainability that are the hallmarks of high-quality software. Consequently, such code can become a source of frustration for developers and may hinder the overall progress of the project. The concept of Technical Debt (TD) was introduced by Cunningham [6], referring to the phenomenon of “not-quite-right code” that represents an incomplete, temporary, or sub-optimal solution.
In the pursuit of short-term advantages, incurring debts, over the 1 Thesis advisor : Dr. Bui Thi Mai Anh Tran Thi Dinh long term, must be repaid at an increasing cost [30]. Self-Admitted Technical Debt (SATD) represents a particular form of TD where devel- opers leave comments within the source code to acknowledge sections that are not fully completed or optimized, indicating the need for further refinement or additional atten- tion [17].1: An example of Self-Admitted Technical Debt of the Defects type in R Figure 1.1 is an example of the SATD categorized into “Defect”. In this example, the comment explicitly mentions the function name, ’validate input’, and the variable ’threshold’, indicating that there is an issue related to the initialization of the variable.
However, it suggests deferring the resolution of this defect, which is a characteristic of the ”Defects” type of SATD. Detecting and effectively dealing with SATD issues presents a multifaceted challenge within the realm of software development. One of the primary complexities lies in the fact that SATD concerns often fly under the radar, remaining obscure and known to only a select few developers. The inherent intricacy of SATD identification and management is compounded by the urgency of the matter.
Failing to address these debts in a timely and efficient manner can lead to a cascading effect of adverse consequences. These repercussions reverberate through various facets of software quality, making it imperative to allocate the requisite time and resources for resolution. In essence, the proactive detection and management of SATD issues are critical com- ponents in the continuous pursuit of software quality enhancement. The intricacies in- volved in this process necessitate a thorough understanding of the various SATD types, as well as the adoption of effective strategies for their detection and subsequent mitiga- tion.
By doing so, software development teams can ensure that their projects remain on a trajectory towards improved reliability, maintainability, and overall excellence. While Self-Admitted Technical Debt (SATD) may not lead to traditional ”disasters” in the sense of natural disasters, it can have significant negative impacts on software projects, 2 Thesis advisor : Dr. Bui Thi Mai Anh Tran Thi Dinh organizations, and even individuals. SATD can lead to software failures, causing applica- tions to crash or malfunction.
For instance, in 2012, Knight Capital Group lost 440 million dolars in under an hour due to a trading algorithm error caused by an undetected software bug, which can be considered a financial disaster.2: Real-world scenarios where SATD has caused substantial problems Technical debt often includes poor security practices. Unresolved security issues in software can lead to data breaches, exposing sensitive information. The Equifax data breach in 2017, affecting 147 million people, was partly attributed to unpatched software vulnerabilities. As a multi-paradigm programming language, R is being used more and more in ap- plications related to data science and statistics [33].
Despite the fact that the R end-user programming community has been growing, the bulk of contributors are statisticians and scientists rather than software engineers. Indeed, prior research has indicated that few R-users are familiar with the nuances of the programming language and that they do not view themselves as developers [26]. While there are several studies dealing with SATD in programming languages such as Java [30], little attention has been paid to the detection of technical debt in the R language. We hypothesize that the detection of SATD in R pack- ages is more difficult than that in other languages due to the mixing of several program- ming paradigms including dynamically typing.
In the context of Java, which is widely used in various software projects, SATD detection becomes particularly relevant. SATD in Java refers to code segments where developers have knowingly introduced suboptimal solutions, which are often documented as comments within the code.Java codebases are typically large and complex, making it challenging to manually identify SATD.
