What Were You Thinking? An LLM-Driven Large-Scale Study of Refactoring Motivations in Open-Source Projects
Top Cited Papers
Open Access
- 28 February 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
Abstract
Context . Code refactoring improves software quality without changing external behavior. Despite its advantages, its benefits are hindered by the considerable cost of time, resources, and continuous effort it demands. Aim . Understanding why developers refactor, and which metrics capture these motivations, may support wider and more effective use of refactoring in practice. Method . We performed a large-scale empirical study to analyze developers’ refactoring activity, leveraging Large Language Models (LLMs) to identify underlying motivations from version control data, comparing our findings with previous motivations reported in the literature. Results . LLMs matched human judgment in 80% of cases, but aligned with literature-based motivations in only 47%. They enriched 22% of motivations with more detailed rationale, often highlighting readability, clarity, and structural improvements. Most motivations were pragmatic, focused on simplification and maintainability. While metrics related to developer experience and code readability ranked highest, their correlation with motivation categories was weak. Conclusions . We conclude that LLMs effectively capture surface-level motivations but struggle with architectural reasoning. Their value lies in providing localized explanations, which, when combined with software metrics, can form hybrid approaches. Such integration offers a promising path toward prioritizing refactoring more systematically and balancing short-term improvements with long-term architectural goals.Keywords
All Related Versions
This publication has 84 references indexed in Scilit:
- Exploring the potential of general purpose LLMs in automated software refactoring: an empirical studyAutomated Software Engineering, 2025
- A comprehensive model for code readabilityJournal of Software: Evolution and Process, 2018
- Encyclopedia of Machine Learning and Data Mining2017
- A Case Study of Refactoring Large-Scale Industrial Systems to Efficiently Improve Source Code Quality2014
- Experimentation in Software Engineering2012
- Refactoring: Improving the Design of Existing Code2002
- Convergence of Probability Measures1999
- Sample size in qualitative researchResearch in Nursing & Health, 1995
- Individual Comparisons by Ranking Methods1992
- Note on the Sampling Error of the Difference Between Correlated Proportions or PercentagesPsychometrika, 1947