LLM-based Vulnerability Detection at Project Scale: An Empirical Study
Preprint
Abstract
In this paper, we present the first comprehensive empirical study of specialized LLM-based detectors and compare them with traditional static analyzers at the project scale. Specifically, our study evaluates five latest and representative LLM-based methods and two traditional tools using: 1) an in-house benchmark of 222 known real-world vulnerabilities (C/C++ and Java) to assess detection capability, and 2) 24 active open-source projects, where we manually inspected 385 warnings to assess their practical usability and underlying root causes of failures. Our evaluation yields three key findings: First, while LLM-based detectors exhibit low recall on the in-house benchmark, they still uncover more unique vulnerabilities than traditional tools. Second, in open-source projects, both LLM-based and traditional tools generate substantial warnings but suffer from very high false discovery rates, hindering practical use. Our manual analysis further reveals shallow interprocedural reasoning and misidentified source/sink pairs as primary failure causes, with LLM-based tools exhibiting additional unique failures. Finally, LLM-based methods incurs substantial computational costs-hundreds of thousands to hundreds of millions of tokens and multi-hour to multi-day runtimes. Overall, our findings underscore critical limitations in the robustness, reliability, and scalability of current LLM-based detectors. We ultimately summarize a set of implications for future research toward more effective and practical project-scale vulnerability detection.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability DetectionProceedings of the ACM on Software Engineering, 2025
- RepairAgent: An Autonomous, LLM-Based Agent for Program Repair2025
- StagedVulBERT: Multigranular Vulnerability Detection With a Novel Pretrained Code ModelIEEE Transactions on Software Engineering, 2024
- Automatic Semantic Augmentation of Language Model Prompts (for Code Summarization)2024
- GRACE: Empowering LLM-based software vulnerability detection with graph structure and in-context learningJournal of Systems and Software, 2024
- LLMDFA: Analyzing Dataflow in Code with Large Language Models2024
- Learning to Detect Memory-related VulnerabilitiesACM Transactions on Software Engineering and Methodology, 2023
- DeepWukongACM Transactions on Software Engineering and Methodology, 2021
- The Art, Science, and Engineering of Fuzzing: A SurveyIEEE Transactions on Software Engineering, 2019
- Static analysis for securityIEEE Security & Privacy, 2004