One Adapter for All Programming Languages? Adapter Tuning for Code Search and Summarization
Authors:
- 1 May 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5-16
Abstract
As pre-trained models automate many code intel-ligence tasks, a widely used paradigm is to fine-tune a model on the task dataset for each programming language. A recent study reported that multilingual fine-tuning benefits a range of tasks and models. However, we find that multilingual fine-tuning leads to performance degradation on recent models UniXcoder and CodeT5. To alleviate the potentially catastrophic forgetting issue in multilingual models, we fix all pre-trained model parameters, insert the parameter-efficient structure adapter, and fine-tune it. Updating only 0.6% of the overall parameters compared to full-model fine-tuning for each programming language, adapter tuning yields consistent improvements on code search and sum-marization tasks, achieving state-of-the-art results. In addition, we experimentally show its effectiveness in cross-lingual and low-resource scenarios. Multilingual fine-tuning with 200 samples per programming language approaches the results fine-tuned with the entire dataset on code summarization. Our experiments on three probing tasks show that adapter tuning significantly outperforms full-model fine-tuning and effectively overcomes catastrophic forgetting.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62032019,61872373,62272473)
This publication has 18 references indexed in Scilit:
- Bridging pre-trained models and downstream tasks for source code understanding2022
- What do pre-trained code models know about code?2021
- IntelliCode compose: code generation using transformer2020
- Modular Tree Network for Source Code Representation LearningACM Transactions on Software Engineering and Methodology, 2020
- BugSum2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Improving automatic source code summarization via deep reinforcement learning2018
- Deep code search2018
- Bug Localization with Combination of Deep Learning and Information Retrieval2017
- Catastrophic forgetting in connectionist networksTrends in Cognitive Sciences, 1999