Long-term Static Analysis Rule Quality Monitoring Using True Negatives
Authors:
- 1 May 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 315-326
Abstract
Static application security testing (SAST) tools have found broad adoption in modern software development work-flows. These tools employ a variety of static analysis rules to generate recommendations on how to improve the code of an application.Every recommendation consumes the time of the engineer that is investigating it, so it is important to measure how useful these rules are in the long term. But what is a good metric for monitoring rule quality over time? Counting the number of recommendations rewards noisy rules and ignores developers’ reactions. Measuring fix rate is not ideal either, because it over-emphasizes rules that are easy to fix.In this paper, we report on an experiment where we use the frequency of true negatives to quantify if developers are able to learn a static analysis rule. We consider a static analysis rule to be ideal if its recommendations are not only addressed, but also internalized by the developer in a way that prevents the bug from recurring. That is, the rule contributes to code quality not only at present, but also in the future. We measure how often developers produce true negatives, that is, code changes that are relevant to a rule but do not trigger a recommendation, and we compare true-negative rate against other metrics. Our results show that measuring true negatives provides insights that cannot be provided by metrics such as fix rate or developer feedback.Keywords
This publication has 12 references indexed in Scilit:
- How should compilers explain problems to developers?2018
- From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis2018
- Lessons from building static analysis tools at GoogleCommunications of the ACM, 2018
- How Developers Diagnose Potential Security Vulnerabilities with a Static Analysis ToolIEEE Transactions on Software Engineering, 2018
- What developers want and need from program analysis: an empirical study2016
- Tracking Static Analysis Violations over Time to Capture Developer Characteristics2015
- A few billion lines of code laterCommunications of the ACM, 2010
- Which warnings should I fix first?2007
- Patches as better bug reports2006
- Tracking defect warnings across versions2006