Evaluating the Generalizability of LLMs in Automated Program Repair
Authors:
- 27 April 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 91-95
Abstract
LLM-based automated program repair methods have attracted significant attention for their state-of-the-art performance. However, they were primarily evaluated on a few well-known datasets like Defects4J, raising questions about their effectiveness on new datasets. In this study, we evaluate 11 top-performing LLMs on DEFECTS4J-TRANS, a new dataset derived from transforming Defects4J while maintaining the original semantics. Results from experiments on both Defects4J and DEFECTS4J-TRANS show that all studied LLMs have limited generalizability in APR tasks, with the average number of correct and plausible patches decreasing by 49.48% and 42.90%, respectively, on DEFECTS4J-TRANS. Further investigation into incorporating additional repair-relevant information in repair prompts reveals that, although this information significantly enhances the LLMs’ capabilities (increasing the number of correct and plausible patches by up to 136.67% and 121.82%, respectively), performance still falls short of their original results. This indicates that prompt engineering alone is insufficient to substantially enhance LLMs’ repair capabilities. Based on our study, we also offer several recommendations for future research.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China
This publication has 24 references indexed in Scilit:
- The rise and potential of large language model based agents: a surveyScience China Information Sciences, 2025
- Gamma: Revisiting Template-Based Automated Program Repair Via Mask Prediction2023
- The Plastic Surgery Hypothesis in the Era of Large Language Models2023
- Code Difference Guided Adversarial Example Generation for Deep Code Models2023
- Tare: Type-Aware Neural Program Repair2023
- Impact of Code Language Models on Automated Program Repair2023
- Automated Program Repair in the Era of Large Pre-trained Language Models2023
- KNOD: Domain Knowledge Distilled Tree Decoder for Automated Program Repair2023
- Inferring Program Transformations From Singular Examples via Big Code2019
- SPPlagiarise: A Tool for Generating Simulated Semantics-Preserving Plagiarism of Java Source Code2019