Towards Understanding the Characteristics of Code Generation Errors Made by Large Language Models
Authors:
Top Cited Papers
- 26 April 2025
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2587-2599
Abstract
Large Language Models (LLMs) have demonstrated unprecedented capabilities in code generation. However, there remains a limited understanding of code generation errors that LLMs can produce. To bridge the gap, we conducted an in-depth analysis of code generation errors across six representative LLMs on the HumanEval dataset. Specifically, we first employed open coding and thematic analysis to distill a comprehensive taxonomy of code generation errors. We analyzed two dimensions of error characteristics-semantic characteristics and syntactic characteristics. Our analysis revealed that LLMs often made non-trivial, multi-line code generation errors in various locations and with various root causes. We further analyzed the correlation between these errors and task complexity as well as test pass rate. Our findings highlighted several challenges in locating and fixing code generation errors made by LLMs. In the end, we discussed several future directions to address these challenges.Keywords
All Related Versions
This publication has 56 references indexed in Scilit:
- What Makes Good In-Context Demonstrations for Code Intelligence Tasks with LLMs?2023
- Come for syntax, stay for speed, understand defects: an empirical study of defects in Julia programsEmpirical Software Engineering, 2023
- Fault Localization with Code Coverage Representation Learning2021
- Harnessing Evolution for Multi-Hunk Program Repair2019
- Precise Condition Synthesis for Program Repair2017
- Codeflaws: a programming competition benchmark for evaluating automated program repair tools2017
- Spectrum-Based Multiple Fault Localization2009
- Toward an understanding of bug fix patternsEmpirical Software Engineering, 2008
- Orthogonal defect classification-a concept for in-process measurementsIEEE Transactions on Software Engineering, 1992
- Measuring nominal scale agreement among many raters.Psychological Bulletin, 1971