A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code
Preprint
Abstract
There is an emerging interest in the application of deep learning models to source code processing tasks. One of the major problems in applying deep learning to software engineering is that source code often contains a lot of rare identifiers resulting in huge vocabularies. We propose a simple yet effective method based on identifier anonymization to handle out-of-vocabulary (OOV) identifiers. Our method can be treated as a preprocessing step and therefore allows an easy implementation. We show that the proposed OOV anonymization method significantly improves the performance of the Transformer in two code processing tasks: code completion and bug fixing.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Empirical study of transformers for source code2021
- Code Prediction by Feeding Trees to Transformers2021
- CodeBERT: A Pre-Trained Model for Programming and Natural Languages2020
- user2code2vec2019
- code2vec: learning distributed representations of codeProceedings of the ACM on Programming Languages, 2019
- Compilation error repair2018
- Self-Attention with Relative Position Representations2018
- Probabilistic model for code with decision trees2016
- Learning programs from noisy data2016
- Morphological Priors for Probabilistic Neural Word Embeddings2016