Is "Better Data" Better than "Better Data Miners"? (On the Benefits of Tuning SMOTE for Defect Prediction)
Preprint
Abstract
We report and fix an important systematic error in prior studies that ranked classifiers for software analytics. Those studies did not (a) assess classifiers on multiple criteria and they did not (b) study how variations in the data affect the results. Hence, this paper applies (a) multi-criteria tests while (b) fixing the weaker regions of the training data (using SMOTUNED, which is a self-tuning version of SMOTE). This approach leads to dramatically large increases in software defect predictions. When applied in a 5*5 cross-validation study for 3,681 JAVA classes (containing over a million lines of code) from open source systems, SMOTUNED increased AUC and recall by 60% and 20% respectively. These improvements are independent of the classifier used to predict for quality. Same kind of pattern (improvement) was observed when a comparative analysis of SMOTE and SMOTUNED was done against the most recent class imbalance technique. In conclusion, for software analytic tasks like defect prediction, (1) data pre-processing can be more important than classifier choice, (2) ranking studies are incomplete without such pre-processing, and (3) SMOTUNED is a promising candidate for pre-processing.Keywords
All Related Versions
This publication has 33 references indexed in Scilit:
- Tuning for software analytics: Is it really necessary?Information and Software Technology, 2016
- Sharing Data and Models in Software Engineering2015
- Software fault prediction metrics: A systematic literature reviewInformation and Software Technology, 2013
- Sample-based software defect prediction with active and semi-supervised learningAutomated Software Engineering, 2011
- An investigation on the feasibility of cross-project defect predictionAutomated Software Engineering, 2011
- Defect prediction from static code features: current results, limitations, new approachesAutomated Software Engineering, 2010
- A systematic review of software fault prediction studiesExpert Systems with Applications, 2009
- Techniques for evaluating fault prediction modelsEmpirical Software Engineering, 2008
- Applying machine learning to software fault-proneness predictionJournal of Systems and Software, 2008
- Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous SpacesJournal of Global Optimization, 1997