Comments on “Researcher Bias: The Use of Machine Learning in Software Defect Prediction”
- 11 April 2016
- journal article
- editorial
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 42 (11) , 1092-1094
Abstract
Shepperd et al. find that the reported performance of a defect prediction model shares a strong relationship with the group of researchers who construct the models. In this paper, we perform an alternative investigation of Shepperd et al.'s data. We observe that (a) research group shares a strong association with other explanatory variables (i.e., the dataset and metric families that are used to build a model); (b) the strong association among these explanatory variables makes it difficult to discern the impact of the research group on model performance; and (c) after mitigating the impact of this strong association, we find that the research group has a smaller impact than the metric family. These observations lead us to conclude that the relationship between the research group and the performance of a defect prediction model are more likely due to the tendency of researchers to reuse experimental components (e.g., datasets and metrics). We recommend that researchers experiment with a broader selection of datasets and metrics to combat any potential bias in their results.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- Views on Internal and External Validity in Empirical Software Engineering2015
- Researcher Bias: The Use of Machine Learning in Software Defect PredictionIEEE Transactions on Software Engineering, 2014
- Ranking and Clustering Software Cost Estimation Models through a Multiple Comparisons AlgorithmIEEE Transactions on Software Engineering, 2012
- Eta squared and partial eta squared as measures of effect size in educational researchEducational Research Review, 2011
- Problems of correlations between explanatory variables in multiple regression analyses in the dental literatureBritish Dental Journal, 2005
- Multicollinearity and Measurement Error in Structural Equation Models: Implications for Theory TestingMarketing Science, 2004
- A new look at the statistical model identificationIEEE Transactions on Automatic Control, 1974