An Empirical Study of False Negatives and Positives of Static Code Analyzers From the Perspective of Historical Issues
Open Access
- 30 September 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
Abstract
Static code analyzers are widely used to help find program flaws. However, in practice the effectiveness and usability of such analyzers is affected by the problems of false negatives (FNs) and false positives (FPs). This paper aims to investigate the FNs and FPs of such analyzers from a new perspective, i.e. , examining the historical issues of FNs and FPs of these analyzers reported by their maintainers, users and researchers in their issue repositories — each of these issues manifested as a FN or FP of these analyzers in the history and has already been confirmed and fixed by the analyzers’ developers. To this end, we conduct the first systematic study on a broad range of 1257 historical issues of FNs/FPs from four popular rule-based static code analyzers for Java ( i.e. , PMD , SpotBugs , SonarQube , and ErrorProne ). All these issues have been confirmed and fixed by the developers. We investigated these issues’ root causes and the characteristics of the corresponding issue-triggering programs. It reveals several new interesting findings and implications on mitigating FNs and FPs. Furthermore, guided by some findings of our study, we designed a metamorphic testing strategy to find FNs and FPs. This strategy successfully found 15 new issues of FNs/FPs, 12 of which have been confirmed and 9 have already been fixed by the developers. Our further manual investigation of the studied analyzers revealed one rule specification issue and additional three FNs/FPs due to the weaknesses of the implemented static analysis. We have made all the artifacts (datasets and tools) publicly available at https://zenodo.org/doi/10.5281/zenodo.11525129 .Keywords
All Related Versions
This publication has 47 references indexed in Scilit:
- Can static analysis tools find more defects?Empirical Software Engineering, 2022
- An Empirical Study of Bugs in WebAssembly Compilers2021
- On the Real-World Effectiveness of Static Bug Detectors at Finding Null Pointer Exceptions2021
- SpongeBugs: Automatically generating fix suggestions in response to static code analysis warningsJournal of Systems and Software, 2020
- How developers engage with static analysis tools in different contextsEmpirical Software Engineering, 2019
- Automatic testing of symbolic execution engines via program generation and differential testing2017
- To what extent could we detect field defects? An extended empirical study of false negatives in static bug-finding toolsAutomated Software Engineering, 2014
- Testing Static Analyzers with Randomly Generated Programs2012
- Static correspondence and correlation between field defects and warnings reported by a bug finding toolSoftware Quality Journal, 2011
- Comparison of classification accuracy using Cohen’s Weighted KappaExpert Systems with Applications, 2008