CodeEditor : Learning to Edit Source Code with Pre-trained Models
Open Access
- 30 September 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 32 (6) , 1-22
Abstract
Developers often perform repetitive code editing activities (up to 70%) for various reasons (e.g., code refactoring) during software development. Many deep learning (DL) models have been proposed to automate code editing by learning from the code editing history. Among DL-based models, pre-trained code editing models have achieved the state-of-the-art (SOTA) results. Pre-trained models are first pre-trained with pre-training tasks and fine-tuned with the code editing task. Existing pre-training tasks mainly are code infilling tasks (e.g., masked language modeling), which are derived from the natural language processing field and are not designed for automatic code editing. In this paper, we propose a novel pre-training task specialized in code editing and present an effective pre-trained code editing model named CodeEditor. Compared to previous code infilling tasks, our pre-training task further improves the performance and generalization ability of code editing models. Specifically, we collect lots of real-world code snippets as the ground truth and use a powerful generator to rewrite them into mutated versions. Then, we pre-train our CodeEditor to edit mutated versions into the corresponding ground truth, to learn edit patterns. We conduct experiments on four code editing datasets and evaluate the pre-trained CodeEditor in three settings (i.e., fine-tuning, few-shot, and zero-shot). (1) In the fine-tuning setting, we train the pre-trained CodeEditor with four datasets and evaluate it on the test data. CodeEditor outperforms the SOTA baselines by 15%, 25.5%, and 9.4% and 26.6% on four datasets. (2) In the few-shot setting, we train the pre-trained CodeEditor with limited data and evaluate it on the test data. CodeEditor substantially performs better than all baselines, even outperforming baselines that are fine-tuned with all data. (3) In the zero-shot setting, we evaluate the pre-trained CodeEditor on the test data without training. CodeEditor correctly edits 1,113 programs while the SOTA baselines can not work. The results show that the superiority of our pre-training task and the pre-trained CodeEditor is more effective in automatic code editing.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62192733, 62192731, 61751210, 62072007, 61832009, 62192730)
This publication has 16 references indexed in Scilit:
- Multi-task learning based pre-trained language model for code completion2020
- CODIT: Code Editing With Tree-Based Neural ModelsIEEE Transactions on Software Engineering, 2020
- TreeGen: A Tree-Based Transformer Architecture for Code GenerationProceedings of the AAAI Conference on Artificial Intelligence, 2020
- CodeBERT: A Pre-Trained Model for Programming and Natural Languages2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- On Learning Meaningful Code Changes Via Neural Machine Translation2019
- Energy and Policy Considerations for Deep Learning in NLP2019
- API code recommendation using statistical learning from fine-grained changes2016
- Neural Machine Translation of Rare Words with Subword Units2016
- Impact of Peer Code Review on Peer Impression Formation: A Survey2013