Demystifying Memorization in LLM-Based Program Repair via a General Hypothesis Testing Framework
Open Access
- 19 June 2025
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 2 (FSE) , 2712-2734
Abstract
Large Language Models (LLMs) have achieved remarkable success in various applications, particularly in code-related tasks such as code generation and program repair, setting new performance benchmarks. However, the extensive use of large training corpora raises concerns about whether these achievements stem from genuine understanding or mere memorization of training data—a question often overlooked in current research. This paper aims to study the memorization issue within LLM-based program repair by investigating whether the correct patches generated by LLMs are the result of memorization. The key challenge lies in the absence of ground truth for confirming memorization, leading to various ad-hoc methods designed for its detection. To address this challenge, we first propose a general framework that formalizes memorization detection as a general hypothesis testing problem, where existing approaches can be unified by defining a low-probability event under the null hypothesis that the data is not memorized. The occurrence of such an event leads to the rejection of the null hypothesis, indicating potential memorization. Based on this framework, we design two specific methods (i.e., low-probability events) to detect potential memorization: 1) basic ground-truth matching, and 2) reassessment after substantial code mutation. We investigate the memorization issue in LLM-based program repair using two datasets: Defects4J, a widely used benchmark that is likely included in the training data, and GitBug-Java, a new dataset that is unlikely to be part of the training data. Our findings reveal that a significant portion of correct patches exactly match the ground truths in Defects4J (e.g., 78.83% and 87.42% on GPT-3.5 and CodeLlama-7b, respectively). Moreover, even after significant modifications to the buggy code, where the original repairs should not be generated, a considerable percentage of bugs (e.g., 81.82% on GPT-3.5 and 88.24% on CodeLlama-7b) continue to be fixed exactly as in the original bug fixes, indicating a high likelihood of memorization. Furthermore, we evaluate existing memorization detection methods and demonstrate their ineffectiveness in this context (e.g., most AUROCs are below 0.5). The theoretical analysis under our hypothesis testing framework shows that their defined events may not meet the requirements for being low-probability. The study highlights the critical need for more robust and rigorous evaluations in LLM-based software engineering research, ensuring a clear distinction between true problem-solving capabilities and mere memorization.Keywords
This publication has 31 references indexed in Scilit:
- LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical EvaluationIEEE Transactions on Software Engineering, 2024
- Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Models2024
- Gamma: Revisiting Template-Based Automated Program Repair Via Mask Prediction2023
- Automated Repair of Programs from Large Language Models2023
- Membership Inference Attacks against Language Models via Neighbourhood Comparison2023
- Memorization and generalization in neural code intelligence modelsInformation and Software Technology, 2022
- The Development of Modus Ponens in Antiquity : From Aristotle to the 2nd Century ADPhronesis, 2002
- An empirical study of smoothing techniques for language modelingComputer Speech & Language, 1999
- Insurance for Low-Probability Hazards: A Bimodal Response to Unlikely Events1993
- Decision Processes for Low Probability Events: Policy ImplicationsJournal of Policy Analysis and Management, 1989