Understanding Industry Perspectives of Static Application Security Testing (SAST) Evaluation
Open Access
- 19 June 2025
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 2 (FSE) , 3033-3056
Abstract
The demand for automated security analysis techniques, specifically static application security testing (SAST), is steadily rising. Assessing the effectiveness of SAST tools is crucial for evaluating current techniques and inspiring future technical advancements. Regrettably, recent research suggests that existing benchmarks used for evaluation often fail to meet the industry's needs, significantly impeding the adoption of SASTs in real-world scenarios. This paper presents a qualitative study to bridge this gap. We investigate why industrial professionals utilize SAST benchmarks, identify barriers to their usage, and explore potential improvements for existing benchmarks. Specifically, we conducted in-depth, semi-structured interviews with twenty industrial professionals possessing diverse field experience and backgrounds in security and product development. As the first comprehensive investigation of SAST evaluation from an industrial perspective, our findings would break down the barriers between academia and industry, providing valuable inspiration for designing better benchmarks and promoting new advances in SAST evaluation.Keywords
Funding Information
- National Natural Science Foundation of China (Grant No. 62302434, U2341212, 62302442)
- Leading Innovative and Entrepreneur Team Introduction Program of Hangzhou (Grant No. TD2020001)
This publication has 72 references indexed in Scilit:
- BUGSC++: A Highly Usable Real World Defect Benchmark for C/C++2023
- A systematic literature review on benchmarks for evaluating debugging approachesJournal of Systems and Software, 2022
- TaintBench: Automatic real-world malware benchmarking of Android taint analysesEmpirical Software Engineering, 2021
- Defects4J as a Challenge Case for the Search-Based Software Engineering Community2020
- Ethical Interviews in Software Engineering2019
- Benchmark Requirements for Assessing Software Security Vulnerability Testing Tools2018
- Static analysis of android apps: A systematic literature reviewInformation and Software Technology, 2017
- Analysing Data2016
- Conducting Semi‐Structured Interviews2015
- Analyzing regression test selection techniquesIEEE Transactions on Software Engineering, 1996