An Empirical Study on Static Application Security Testing (SAST) Tools for Python
Open Access
- 12 April 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 3673-3685
Abstract
Python is currently the most popular programming language and ensuring the security of Python applications has become a critical concern. Static Application Security Testing (SAST) tools have been introduced to address this need, claiming to support a wide range of Common Weakness Enumerations (CWEs). However, the ability of these tools to detect real-world vulnerabilities in Python programs has not been comprehensively evaluated. In this paper, we selected eight SAST tools from 117 existing ones based on well-designed criteria. Based on the synthetic and real-world dataset, we evaluated and compared these SAST tools from different perspectives including effectiveness and efficiency. Our results reveal significant limitations in current SAST tools: although perform well on the synthetic dataset, a single tool detects no more than 40% of the vulnerabilities in our real-world dataset. Even when aggregating the outputs of all evaluated tools, only 66.7% of the real-world vulnerabilities are identified. To further understand these shortcomings, we performed a root cause analysis of the detection results and identified useful insights for both SAST tool designers and users, focusing on tool development, evaluation, and selection.This publication has 27 references indexed in Scilit:
- Empirical analysis of security vulnerabilities in Python packagesEmpirical Software Engineering, 2023
- On the adoption of static analysis for software security assessment–A case study of an open-source e-government projectComputers & Security, 2021
- TaintBench: Automatic real-world malware benchmarking of Android taint analysesEmpirical Software Engineering, 2021
- On the Use of Open-Source C/C++ Static Analysis Tools in Large Projects2020
- Challenges with Responding to Static Analysis Tool Alerts2019
- An empirical study on combining diverse static analysis tools for web security vulnerabilities based on development scenariosComputing, 2018
- Context is king: The developer perspective on the usage of static analysis tools2018
- Comparative study on static code analysis tools for C/C++2018
- Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software2016
- Static Analysis: A Survey of Techniques and Tools2015