An empirical evaluation of GitHub copilot's code suggestions
Authors:
Top Cited Papers
Open Access
- 23 May 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-5
Abstract
GitHub and OpenAI recently launched Copilot, an "AI pair programmer" that utilizes the power of Natural Language Processing, Static Analysis, Code Synthesis, and Artificial Intelligence. Given a natural language description of the target functionality, Copilot can generate corresponding code in several programming languages. In this paper, we perform an empirical study to evaluate the correctness and understandability of Copilot's suggested code. We use 33 LeetCode questions to create queries for Copilot in four different programming languages. We evaluate the correctness of the corresponding 132 Copilot solutions by running LeetCode's provided tests, and evaluate understandability using SonarQube's cyclomatic complexity and cognitive complexity metrics. We find that Copilot's Java suggestions have the highest correctness score (57%) while JavaScript is the lowest (27%). Overall, Copilot's suggestions have low complexity with no notable differences between the programming languages. We also find some potential Copilot shortcomings, such as generating code that can be further simplified and code that relies on undefined helper methods.Keywords
Funding Information
- Canada Research Chairs Program
This publication has 12 references indexed in Scilit:
- Aroma: code recommendation via structural code searchProceedings of the ACM on Programming Languages, 2019
- When deep learning met code search2019
- CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning2019
- PeerLens2019
- code2vec: learning distributed representations of codeProceedings of the ACM on Programming Languages, 2019
- A Survey of Machine Learning for Big Code and NaturalnessACM Computing Surveys, 2018
- Code Completion with Neural Attention and Pointer Networks2018
- Cognitive complexity2018
- Deep code search2018
- On the naturalness of software2012