Diet code is healthy: simplifying programs for pre-trained models of code
Authors:
Open Access
- 7 November 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1073-1084
Abstract
Pre-trained code representation models such as CodeBERT have demonstrated superior performance in a variety of software engineering tasks, yet they are often heavy in complexity, quadratically with the length of the input sequence. Our empirical analysis of CodeBERT's attention reveals that CodeBERT pays more attention to certain types of tokens and statements such as keywords and data-relevant statements. Based on these findings, we propose DietCode, which aims at lightweight leverage of large pre-trained models for source code. DietCode simplifies the input program of CodeBERT with three strategies, namely, word dropout, frequency filtering, and an attention-based strategy that selects statements and tokens that receive the most attention weights during pre-training. Hence, it gives a substantial reduction in the computational cost without hampering the model performance. Experimental results on two downstream tasks show that DietCode provides comparable results to CodeBERT with 40% less computational cost in fine-tuning and testing.Keywords
All Related Versions
Funding Information
- NSF (National Science Foundation) (62102244)
- CCF-Baidu Open Fund (2021PP15002000)
- CCF-Tencent Rhino-Bird Young Faculty Open Research Fund ()
This publication has 26 references indexed in Scilit:
- Model Compression and Acceleration for Deep Neural Networks: The Principles, Progress, and ChallengesIEEE Signal Processing Magazine, 2018
- Deep learning code fragments for code clone detection2016
- Query Expansion Based on Crowd Knowledge for Code SearchIEEE Transactions on Services Computing, 2016
- Deep Unordered Composition Rivals Syntactic Methods for Text Classification2015
- Improving automated source code summarization via an eye-tracking study of programmers2014
- Solving the Search for Source CodeACM Transactions on Software Engineering and Methodology, 2014
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- On the naturalness of software2012
- CCFinder: a multilinguistic token-based code clone detection system for large scale source codeIEEE Transactions on Software Engineering, 2002
- Simplifying and isolating failure-inducing inputIEEE Transactions on Software Engineering, 2002