An Empirical Study of Model-Agnostic Techniques for Defect Prediction Models
Top Cited Papers
- 23 March 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 48 (1) , 166-185
Abstract
Software analytics have empowered software organisations to support a wide range of improved decision-making and policy-making. However, such predictions made by software analytics to date have not been explained and justified. Specifically, current defect prediction models still fail to explain why models make such a prediction and fail to uphold the privacy laws in terms of the requirement to explain any decision made by an algorithm. In this paper, we empirically evaluate three model-agnostic techniques, i.e., two state-of-the-art Local Interpretability Model-agnostic Explanations technique (LIME) and BreakDown techniques, and our improvement of LIME with Hyper Parameter Optimisation (LIME-HPO). Through a case study of 32 highly-curated defect datasets that span across 9 open-source software systems, we conclude that (1) model-agnostic techniques are needed to explain individual predictions of defect models; (2) instance explanations generated by model-agnostic techniques are mostly overlapping (but not exactly the same) with the global explanation of defect models and reliable when they are re-generated; (3) model-agnostic techniques take less than a minute to generate instance explanations; and (4) more than half of the practitioners perceive that the contrastive explanations are necessary and useful to understand the predictions of defect models. Since the implementation of the studied model-agnostic techniques is available in both Python and R, we recommend model-agnostic techniques be used in the future.Keywords
Funding Information
- Australian Research Council (DE200100941)
- ARC Laureate Fellowship (FL190100035)
This publication has 88 references indexed in Scilit:
- Gradient boosting machines, a tutorialFrontiers in Neurorobotics, 2013
- Special issue on repeatable results in software engineering predictionEmpirical Software Engineering, 2012
- Evaluating defect prediction approaches: a benchmark and an extensive comparisonEmpirical Software Engineering, 2011
- A systematic and comprehensive investigation of methods to build and evaluate fault prediction modelsJournal of Systems and Software, 2010
- The WEKA data mining softwareACM SIGKDD Explorations Newsletter, 2009
- Conditional variable importance for random forestsBMC Bioinformatics, 2008
- Predicting defect-prone software modules using support vector machinesJournal of Systems and Software, 2008
- Agile software development: Human values and cultureIEEE Technology and Society Magazine, 2005
- Remote Causes, Bad Explanations?Journal for the Theory of Social Behaviour, 2002
- SURVEY RESEARCHAnnual Review of Psychology, 1999