Validating Unsupervised Machine Learning Techniques for Software Defect Prediction With Generic Metamorphic Testing
Open Access
- 8 November 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 165155-165172
Abstract
In the realm of software defect prediction, unsupervised models often step in when labelled datasets are scarce, despite facing the challenge of validating models without prior knowledge of data. Addressing this, we proposed a novel approach leveraging generic metamorphic testing to validate such models effectively, bypassing the need for expert-derived metamorphic relations. Our method identifies five categories of generic metamorphic relations, further divided into twenty-one individual generic metamorphic relations, all formulated through generic Data Mutation Operators. This framework enables us to generate follow-up datasets from the source datasets, training respective software defect prediction models. By comparing predictions between the source and follow-up software defect prediction models and identifying inconsistencies, we can assess the model’s sensitivity to generic metamorphic relations as a form of validation. This approach was rigorously evaluated across twenty software defect prediction models, incorporating fourteen different machine learning algorithms and twenty high-dimensional public datasets. Remarkably, the robustness of our generic MT model was confirmed, showing substantial effectiveness in validating software defect prediction models, independent of whether they were supervised or unsupervised. Software defect prediction models, using Agglomerative clustering and Density-Based Spatial Clustering of Applications with Noise algorithms, did not violate any metamorphic relation, and nineteen software defect prediction models did not significantly violate the generic metamorphic relation "Shrinkage and Expansion". Our findings suggest that employing generic metamorphic relations, especially "Shrinkage and Expansion", can universally enhance the validation of defect prediction models. Furthermore, our model can be applied to continuously monitor software defect prediction models.Keywords
Funding Information
- General Research Fund of the Research Grants Council of Hong Kong
- research funds of the City University of Hong Kong (6000796)
This publication has 50 references indexed in Scilit:
- Smoke testing for machine learning: simple tests to discover severe bugsEmpirical Software Engineering, 2022
- Verification of the neural network training process for spectrum-based chemical substructure prediction using metamorphic testingJournal of Computational Science, 2021
- Validating class integration test order generation systems with Metamorphic TestingInformation and Software Technology, 2021
- A comprehensive comparative study of clustering-based unsupervised defect prediction modelsJournal of Systems and Software, 2021
- A survey of density based clustering algorithmsFrontiers of Computer Science, 2020
- Testing machine learning based systems: a systematic mappingEmpirical Software Engineering, 2020
- A systematic review of unsupervised learning techniques for software defect predictionInformation and Software Technology, 2020
- Software defect number prediction: Unsupervised vs supervised methodsInformation and Software Technology, 2019
- A Comprehensive Survey of Clustering AlgorithmsAnnals of Data Science, 2015
- Testing and validating machine learning classifiers by metamorphic testingJournal of Systems and Software, 2011