Big code != big vocabulary
Authors:
Open Access
- 27 June 2020
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1073-1085
Abstract
Statistical language modeling techniques have successfully been applied to large source code corpora, yielding a variety of new software development tools, such as tools for code suggestion, improving readability, and API migration. A major issue with these techniques is that code introduces new vocabulary at a far higher rate than natural language, as new identifier names proliferate. Both large vocabularies and out-of-vocabulary issues severely affect Neural Language Models (NLMs) of source code, degrading their performance and rendering them unable to scale. In this paper, we address this issue by: 1) studying how various modelling choices impact the resulting vocabulary on a large-scale corpus of 13,362 projects; 2) presenting an open vocabulary source code NLM that can scale to such a corpus, 100 times larger than in previous work; and 3) showing that such models outperform the state of the art on three distinct code corpora (Java, C, Python). To our knowledge, these are the largest NLMs for code that have been reported. All datasets, code, and trained models used in this work are publicly available.Keywords
All Related Versions
Funding Information
- ADVERB
- IDEALS
- Engineering and Physical Sciences Research Council (,EP/L016427/1)
This publication has 57 references indexed in Scilit:
- Suggesting accurate method and class names2015
- CACHECA: A Cache Language Model Based Code Suggestion Tool2015
- Toward Deep Learning Software Repositories2015
- On Using Very Large Target Vocabulary for Neural Machine Translation2015
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- An empirical study of identifier splitting techniquesEmpirical Software Engineering, 2013
- Learning from examples to improve code completion systems2009
- The Porter stemming algorithm: then and nowProgram: electronic library and information systems, 2006
- An empirical study of smoothing techniques for language modelingComputer Speech & Language, 1999
- Long Short-Term MemoryNeural Computation, 1997