Evaluating Large Language Models for Line-Level Vulnerability Localization
- 30 December 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 52 (3) , 770-785
Abstract
Recently, Automated Vulnerability Localization (AVL) has attracted growing attention, aiming to facilitate diagnosis by pinpointing the specific lines of code responsible for vulnerabilities. Large Language Models (LLMs) have shown potential in various domains, yet their effectiveness in line-level vulnerability localization remains underexplored. In this work, we present the first comprehensive empirical evaluation of LLMs for AVL. Our study examines 19 leading LLMs suitable for code analysis, including ChatGPT and multiple open-source models, spanning encoder-only, encoder-decoder, and decoder-only architectures, with model sizes from 60M to 70B parameters. We evaluate three paradigms—few-shot prompting, discriminative fine-tuning, and generative fine-tuning—with and without Low-Rank Adaptation (LoRA), on both a BigVul-derived dataset for C/C++ and a smart contract vulnerability dataset. Our results show that discriminative fine-tuning achieves substantial performance gains over existing learning-based AVL methods when sufficient training data is available. In low-data settings, prompting advanced LLMs such as ChatGPT proves more effective. We also identify challenges related to input length and unidirectional context during fine-tuning, and propose two remedial strategies: a sliding window approach and right-forward embedding, both of which yield significant improvements. Moreover, we provide the first assessment of LLM generalizability in AVL, showing that certain models can transfer effectively across Common Weakness Enumerations (CWEs) and projects. However, performance degrades notably for newly discovered vulnerabilities containing unfamiliar lexical or structural patterns, underscoring the need for continual adaptation. These findings offer practical guidance for deploying LLM-based AVL systems in realistic software security workflows.Keywords
Funding Information
- National Research Foundation, Singapore, and the DSO National Laboratories under the AI Singapore Programme (AISG4-GC-2023-008-1B)
- National Research Foundation Singapore and the Cyber Security Agency under the National Cybersecurity R&D Programme (NCRP25-P04-TAICeN)
- Prime Minister’s Office, Singapore under the Campus for Research Excellence and Technological Enterprise (CREATE) Programme
- Fundamental Research Funds for the Central Universities (KG16426001)
This publication has 40 references indexed in Scilit:
- When Less is Enough: Positive and Unlabeled Learning Model for Vulnerability Detection2023
- An Empirical Study on Fine-Tuning Large Language Models of Code for Automated Program Repair2023
- Learning to Locate and Describe Vulnerabilities2023
- Data Quality for Software Vulnerability Datasets2023
- An Empirical Comparison of Pre-Trained Models of Source Code2023
- Impact of Code Language Models on Automated Program Repair2023
- Automated Repair of Programs from Large Language Models2023
- Automated Program Repair in the Era of Large Pre-trained Language Models2023
- Fine-grained Commit-level Vulnerability Type Prediction by CWE Tree Structure2023
- A large scale exploratory analysis of software vulnerability life cycles2012