Empirical Evaluation of Large Language Models in Automated Program Repair
Preprint
Abstract
The increasing prevalence of software bugs has made automated program repair (APR) a key research focus. Large language models (LLMs) offer new opportunities for APR, but existing studies mostly rely on smaller, earlier-generation models and Java benchmarks. The repair capabilities of modern, large-scale LLMs across diverse languages and scenarios remain underexplored. To address this, we conduct a comprehensive empirical study of four open-source LLMs, CodeLlama, LLaMA, StarCoder, and DeepSeek-Coder, spanning 7B to 33B parameters, diverse architectures, and purposes. We evaluate them across two bug scenarios (enterprise-grades and algorithmic), three languages (Java, C/C++, Python), and four prompting strategies, analyzing over 600K generated patches on six benchmarks. Key findings include: (1) model specialization (e.g., CodeLlama) can outperform larger general-purpose models (e.g., LLaMA); (2) repair performance does not scale linearly with model size; (3) correct patches often appear early in generation; and (4) prompts significantly affect results. These insights offer practical guidance for designing effective and efficient LLM-based APR systems.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- RepairLLaMA: Efficient Representations and Fine-Tuned Adapters for Program RepairIEEE Transactions on Software Engineering, 2025
- An overview of large AI models and their applicationsVisual Intelligence, 2024
- ConDefects: A Complementary Dataset to Address the Data Leakage Concern for LLM-Based Fault Localization and Program Repair2024
- Hyperparameter Optimization for AST DifferencingIEEE Transactions on Software Engineering, 2023
- Parameter-efficient fine-tuning of large-scale pre-trained language modelsNature Machine Intelligence, 2023
- Visual Prompt Tuning2022
- SEQUENCER: Sequence-to-Sequence Learning for End-to-End Program RepairIEEE Transactions on Software Engineering, 2019
- Nopol: Automatic Repair of Conditional Statement Bugs in Java ProgramsIEEE Transactions on Software Engineering, 2016
- The ManyBugs and IntroClass Benchmarks for Automated Repair of C ProgramsIEEE Transactions on Software Engineering, 2015
- GenProg: A Generic Method for Automatic Software RepairIEEE Transactions on Software Engineering, 2011