On the Generalizability of Deep Learning-based Code Completion Across Programming Language Versions
Open Access
- 15 April 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 99-111
Abstract
Code completion is a key feature of Integrated Development Environments (IDEs), aimed at predicting the next tokens a developer is likely to write, helping them write code faster and with less effort. Modern code completion approaches are often powered by deep learning (DL) models. However, the swift evolution of programming languages poses a critical challenge to the performance of DL-based code completion models: Can these models generalize across different language versions? This paper delves into such a question. In particular, we assess the capabilities of a state-of-the-art model, CodeT5, to generalize across nine different Java versions, ranging from Java 2 to Java 17, while being exclusively trained on Java 8 code. Our evaluation spans three completion scenarios, namely, predicting tokens, constructs (e.g., the condition of an if statement) and entire code blocks. The results of our study reveal a noticeable disparity among language versions, with the worst performance being obtained in Java 2 and 17---the most far apart versions compared to Java 8. We investigate possible causes for the performance degradation and show that the adoption of a limited version-specific fine-tuning can partially alleviate the problem. Our work raises awareness on the importance of continuous model refinement, and it can inform the design of alternatives to make code completion models more robust to language evolution.Keywords
All Related Versions
Funding Information
- European Research Council (851720)
This publication has 47 references indexed in Scilit:
- Source Code Recommender Systems: The Practitioners' Perspective2023
- NatGen: generative pre-training by “naturalizing” source code2022
- AI-Driven Development Is Here: Should You Worry?IEEE Software, 2022
- Learning to Find Usages of Library Functions in Optimized BinariesIEEE Transactions on Software Engineering, 2021
- How we refactor and how we document it? On the use of supervised machine learning algorithms to classify refactoring documentationExpert Systems with Applications, 2021
- Deep Learning Based Program Generation From Requirements Text: Are We There Yet?IEEE Transactions on Software Engineering, 2020
- A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineeringSoftware Testing, Verification and Reliability, 2012
- Towards a better code completion system by API grouping, filtering, and popularity-based ranking2010
- Improving code completion with program historyAutomated Software Engineering, 2010
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple TestingJournal of the Royal Statistical Society Series B: Statistical Methodology, 1995