Empirical Study: How Issue Classification Influences Software Defect Prediction
Open Access
- 3 February 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 11732-11748
Abstract
Software defect prediction aims to identify potentially defective software modules to better allocate limited quality assurance resources. Practitioners often do this by utilizing supervised models trained using historical data. This data is gathered by mining version control and issue tracking systems. Version control commits are linked to issues they address. If the linked issue is classified as a bug report, the change is considered as bug fixing. The problem arises from the fact that issues are often incorrectly classified within issue tracking systems. This introduces noise into the gathered datasets. In this paper, we investigate the influence issue classification has on software defect prediction dataset quality and resulting model performance. To do this, we mine data from 7 popular open-source repositories, create issue classification and software defect prediction datasets for each of them. We investigate issue classification using four different methods; a simple keyword heuristic, an improved keyword heuristic, the FastText model and the RoBERTa model. Our results show that using the RoBERTa model for issue classification produces the best software defect prediction datasets, containing on average 14.3641% of mislabeled instances. SDP models trained on such datasets achieve superior performance, to those trained on SDP datasets created using other issue classification methods, in 65 out of 84 experiments, with 55 of them being statistically relevant. Furthermore, in 17 out of 28 experiments we could not show a statistically relevant performance difference between SDP models trained on RoBERTa derived software defect prediction datasets and those created using manually labeled issues.Keywords
Funding Information
- European Regional Development Fund through the System for Detection of Malicious Transactions in Electronic Payment Operations Based on Machine Learning Research (IRI-II KK.01.2.1.02.0192)
- VODIME—The Waters of Imotski Region Research (KK.05.1.1.02.0024)
- Croatian Science Foundation through the Reliable Composite Applications Based on Web Services Research Project (HRZZ-IP-01-2018-6423)
This publication has 71 references indexed in Scilit:
- The Impact of Mislabelling on the Performance and Interpretation of Defect Prediction Models2015
- A Semi-supervised Approach to Software Defect Prediction2014
- An empirical study of the classification performance of learners on imbalanced and noisy software quality dataInformation Sciences, 2014
- A Convolutional Neural Network for Modelling Sentences2014
- Metrics for measuring the consistencies of requirements with objectives and constraintsRequirements Engineering, 2013
- Student’s t-Tests2011
- Wilcoxon–Mann–Whitney Test2011
- On the relative value of cross-company and within-company data for defect predictionEmpirical Software Engineering, 2009
- Chidamber and Kemerer's metrics suite: a measurement theory perspectiveIEEE Transactions on Software Engineering, 1996
- An algorithm for suffix strippingProgram: electronic library and information systems, 1980