An evaluation of pure spectrum‐based fault localization techniques for large‐scale software systems
Open Access
- 24 May 2019
- journal article
- research article
- Published by Wiley in Software: Practice and Experience
- Vol. 49 (8) , 1197-1224
Abstract
Summary: Pure spectrum‐based fault localization (SBFL) is a well‐studied statistical debugging technique that only takes a set of test cases (some failing and some passing) and their code coverage as input and produces a ranked list of suspicious program elements to help the developer identify the location of a bug that causes a failed test case. Studies show that pure SBFL techniques produce good ranked lists for small programs. However, our previous study based on the iBugs benchmark that uses the AspectJ repository shows that, for realistic programs, the accuracy of the ranked list is not suitable for human developers. In this paper, we confirm this based on a combined empirical evaluation with the iBugs and the Defects4J benchmark. Our experiments show that, on average, at most ∼40%, ∼80%, and ∼90% of the bugs can be localized reliably within the first 10, 100, and 1000 ranked lines, respectively, in the Defects4J benchmark. To reliably localize 90% of the bugs with the best performing SBFL metric D∗, ∼450 lines have to be inspected by the developer. For human developers, this remains unsuitable, although the results improve compared with the results for the AspectJ benchmark. Based on this study, we can clearly see the need to go beyond pure SBFL and take other information, such as information from the bug report or from version history of the code lines, into consideration.Keywords
This publication has 74 references indexed in Scilit:
- A test-suite reduction approach to improving fault-localization effectivenessComputer Languages, Systems & Structures, 2013
- Metallaxis‐FL: mutation‐based fault localizationSoftware Testing, Verification and Reliability, 2013
- Metamorphic slice: An application in spectrum-based fault localizationInformation and Software Technology, 2013
- Context-Aware Fault Localization via Control Flow AnalysisJournal of Software, 2011
- A model for spectra-based software diagnosisACM Transactions on Software Engineering and Methodology, 2011
- Prioritizing tests for software fault diagnosisSoftware: Practice and Experience, 2011
- A practical evaluation of spectrum-based fault localizationJournal of Systems and Software, 2009
- Interactive Fault Localization Using Test InformationJournal of Computer Science and Technology, 2009
- Test input reduction for result inspection to facilitate fault localizationAutomated Software Engineering, 2009
- Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential ImpactEmpirical Software Engineering, 2005