Examination of Code generated by Large Language Models
Authors:
- Alexander FeixAFAlexander Feix
- Welf LöweWelf Löwe
Preprint
Abstract
Large language models (LLMs), such as ChatGPT and Copilot, are transforming software development by automating code generation and, arguably, enable rapid prototyping, support education, and boost productivity. Therefore, correctness and quality of the generated code should be on par with manually written code. To assess the current state of LLMs in generating correct code of high quality, we conducted controlled experiments with ChatGPT and Copilot: we let the LLMs generate simple algorithms in Java and Python along with the corresponding unit tests and assessed the correctness and the quality (coverage) of the generated (test) codes. We observed significant differences between the LLMs, between the languages, between algorithm and test codes, and over time. The present paper reports these results together with the experimental methods allowing repeated and comparable assessments for more algorithms, languages, and LLMs over time.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- What is wrong with your code generated by large language models? An extensive studyScience China Information Sciences, 2026
- LLM-Based Test-Driven Interactive Code Generation: User Study and Empirical EvaluationIEEE Transactions on Software Engineering, 2024
- On Evaluating the Efficiency of Source Code Generated by LLMs2024
- Shortcut Learning of Large Language Models in Natural Language UnderstandingCommunications of the ACM, 2023
- CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X2023
- Competition-level code generation with AlphaCodeScience, 2022
- Assessing the quality of GitHub copilot’s code generation2022
- GPT-3: What’s it good for?Natural Language Engineering, 2020
- Testing Statistical Assumptions in Research2019
- On the naturalness of softwareCommunications of the ACM, 2016