An Empirical Study of Code Smells in Transformer-based Code Generation Techniques
Authors:
Top Cited Papers
- 1 October 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 71-82
Abstract
Prior works have developed transformer-based language learning models to automatically generate source code for a task without compilation errors. The datasets used to train these techniques include samples from open source projects which may not be free of security flaws, code smells, and violations of standard coding practices. Therefore, we investigate to what extent code smells are present in the datasets of coding generation techniques and verify whether they leak into the output of these techniques. To conduct this study, we used Pylint and Bandit to detect code smells and security smells in three widely used training sets (CodeXGlue, APPS, and Code Clippy). We observed that Pylint caught 264 code smell types, whereas Bandit located 44 security smell types in these three datasets used for training code generation techniques. By analyzing the output from ten different configurations of the open-source fine-tuned transformer-based GPT-Neo 125M parameters model, we observed that this model leaked the smells and non-standard practices to the generated source code. When analyzing GitHub Copilot's suggestions, a closed source code generation tool, we observed that it contained 18 types of code smells, including substandard coding patterns and 2 security smell types.Keywords
This publication has 29 references indexed in Scilit:
- Fast and Memory-Efficient Neural Code Completion2021
- Code Prediction by Feeding Trees to Transformers2021
- Code Smells Detection and Visualization: A Systematic Literature ReviewArchives of Computational Methods in Engineering, 2021
- Share, But be Aware: Security Smells in Python Gists2019
- Detecting code smells using machine learning techniques: Are we there yet?2018
- Understanding metric-based detectable smells in Python software: A comparative studyInformation and Software Technology, 2018
- House of Cards: Code Smells in Open-Source C# Repositories2017
- Security Smells in Android2017
- On the diffuseness and the impact on maintainability of code smells: a large scale empirical investigationEmpirical Software Engineering, 2017
- Detecting Code Smells in Python Programs2016