How Do Developers Act on Static Analysis Alerts? An Empirical Study of Coverity Usage
Authors:
- 1 October 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 323-333
Abstract
Static analysis tools (SATs) often fall short of developer satisfaction despite their many benefits. An understanding of how developers in the real-world act on the alerts detected by SATs can help improve the utility of these tools and determine future research directions. The goal of this paper is to aid researchers and tool makers in improving the utility of static analysis tools through an empirical study of developer action on the alerts detected by Coverity, a state-of-the-art static analysis tool. In this paper, we analyze five open source projects as case studies (Linux, Firefox, Samba, Kodi, and Ovirt-engine) that have been actively using Coverity over a period of at least five years. We investigate the alert occurrences and developer triage of the alerts from the Coverity database; identify the alerts that were fixed through code changes (i.e. actionable) by mining the commit history of the projects; analyze the time an alert remain in the code base (i.e. lifespan) and the complexity of code changes (i.e. fix complexity) in fixing the alert. We find that 27.4% to 49.5% (median: 36.7%) of the alerts are actionable across projects, a rate higher than previously reported. We also find that the fixes of Coverity alerts are generally low in complexity (2 to 7 lines of code changes in the affected file, median: 4). However, developers still take from 36 to 245 days (median: 96) to fix these alerts. Finally, our data suggest that severity and fix complexity may correlate with an alert's lifespan in some of the projects.Keywords
This publication has 22 references indexed in Scilit:
- CloudBuild2016
- Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software2016
- Tricorder: Building a Program Analysis Ecosystem2015
- Eliminating Static Analysis False Positives Using Loop Abstraction and Bounded Model Checking2015
- A systematic literature review of actionable alert identification techniques for automated static code analysisInformation and Software Technology, 2011
- Infer: An Automatic Program Verifier for Memory Safety of C Programs2011
- A few billion lines of code laterCommunications of the ACM, 2010
- A Comparative Study of Industrial Static Analysis ToolsElectronic Notes in Theoretical Computer Science, 2008
- Which warnings should I fix first?2007
- Evaluating static analysis defect warnings on production software2007