How Far Have We Progressed in Identifying Self-admitted Technical Debts? A Comprehensive Empirical Study
Authors:
Open Access
- 23 July 2021
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 30 (4) , 1-56
Abstract
Background. Self-admitted technical debt (SATD) is a special kind of technical debt that is intentionally introduced and remarked by code comments. Those technical debts reduce the quality of software and increase the cost of subsequent software maintenance. Therefore, it is necessary to find out and resolve these debts in time. Recently, many automatic approaches have been proposed to identify SATD. Problem. Popular IDEs support a number of predefined task annotation tags for indicating SATD in comments, which have been used in many projects. However, such clear prior knowledge is neglected by existing SATD identification approaches when identifying SATD. Objective. We aim to investigate how far we have really progressed in the field of SATD identification by comparing existing approaches with a simple approach that leverages the predefined task tags to identify SATD. Method. We first propose a simple heuristic approach that fuzzily Matches task Annotation Tags ( MAT ) in comments to identify SATD. In nature, MAT is an unsupervised approach, which does not need any data to train a prediction model and has a good understandability. Then, we examine the real progress in SATD identification by comparing MAT against existing approaches. Result. The experimental results reveal that: (1) MAT has a similar or even superior performance for SATD identification compared with existing approaches, regardless of whether non-effort-aware or effort-aware evaluation indicators are considered; (2) the SATDs (or non-SATDs) correctly identified by existing approaches are highly overlapped with those identified by MAT ; and (3) supervised approaches misclassify many SATDs marked with task tags as non-SATDs, which can be easily corrected by their combinations with MAT . Conclusion. It appears that the problem of SATD identification has been (unintentionally) complicated by our community, i.e., the real progress in SATD comments identification is not being achieved as it might have been envisaged. We hence suggest that, when many task tags are used in the comments of a target project, future SATD identification studies should use MAT as an easy-to-implement baseline to demonstrate the usefulness of any newly proposed approach.Keywords
Funding Information
- National Key Basic Research and Development Program of China (2014CB340702)
- National Natural Science Foundation of China (61772259, 61872177)
This publication has 65 references indexed in Scilit:
- A systematic mapping study on technical debt and its managementJournal of Systems and Software, 2015
- An empirical study of the textual similarity between source code and source code summariesEmpirical Software Engineering, 2014
- An in-depth study of the potentially confounding effect of class size in fault predictionACM Transactions on Software Engineering and Methodology, 2014
- Technical debtACM SIGSOFT Software Engineering Notes, 2013
- Improving program comprehension by combining code understanding with comment understandingKnowledge-Based Systems, 2008
- Machine learning in automated text categorizationACM Computing Surveys, 2002
- The control of the false discovery rate in multiple testing under dependencyThe Annals of Statistics, 2001
- The WyCash portfolio management systemACM SIGPLAN OOPS Messenger, 1992
- A vector space model for automatic indexingCommunications of the ACM, 1975
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960