Benchmarking Large Language Models for Multi-Language Software Vulnerability Detection
Preprint
Abstract
Recent advancements in generative AI have led to the widespread adoption of large language models (LLMs) in software engineering, addressing numerous long-standing challenges. However, a comprehensive study examining the capabilities of LLMs in software vulnerability detection (SVD), a crucial aspect of software security, is currently lacking. Existing research primarily focuses on evaluating LLMs using C/C++ datasets. It typically explores only one or two strategies among prompt engineering, instruction tuning, and sequence classification fine-tuning for open-source LLMs. Consequently, there is a significant knowledge gap regarding the effectiveness of diverse LLMs in detecting vulnerabilities across various programming languages. To address this knowledge gap, we present a comprehensive empirical study evaluating the performance of LLMs on the SVD task. We have compiled a comprehensive dataset comprising 8,260 vulnerable functions in Python, 7,505 in Java, and 28,983 in JavaScript. We assess five open-source LLMs using multiple approaches, including prompt engineering, instruction tuning, and sequence classification fine-tuning. These LLMs are benchmarked against five fine-tuned small language models and two open-source static application security testing tools. Furthermore, we explore two avenues to improve LLM performance on SVD: a) Data perspective: Retraining models using downsampled balanced datasets. b) Model perspective: Investigating ensemble learning methods that combine predictions from multiple LLMs. Our comprehensive experiments demonstrate that SVD remains a challenging task for LLMs. This study provides a thorough understanding of the role of LLMs in SVD and offers practical insights for future advancements in leveraging generative AI to enhance software security practices.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- Empirical studies of parameter efficient methods for large language models of code and knowledge transfer to REmpirical Software Engineering, 2025
- Finetuning Large Language Models for Vulnerability DetectionIEEE Access, 2025
- Utilization of pre-trained language models for adapter-based knowledge transfer in software engineeringEmpirical Software Engineering, 2024
- GRACE: Empowering LLM-based software vulnerability detection with graph structure and in-context learningJournal of Systems and Software, 2024
- An Empirical Comparison of Pre-Trained Models of Source Code2023
- Parameter-efficient fine-tuning of large-scale pre-trained language modelsNature Machine Intelligence, 2023
- SySeVR: A Framework for Using Deep Learning to Detect Software VulnerabilitiesIEEE Transactions on Dependable and Secure Computing, 2021
- Software Vulnerability Discovery via Learning Multi-Domain Knowledge BasesIEEE Transactions on Dependable and Secure Computing, 2019
- Detecting and Preventing Cyber Insider Threats: A SurveyIEEE Communications Surveys & Tutorials, 2018
- ELBlocker: Predicting blocking bugs with ensemble imbalance learningInformation and Software Technology, 2015