Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code
Authors:
- Rafael-Michael KarampatsisRKRafael-Michael Karampatsis
- Hlib BabiiHBHlib Babii
- Romain RobbesRRRomain Robbes
- Charles SuttonCSCharles Sutton
- Andrea JanesAJAndrea Janes
Preprint
Abstract
Statistical language modeling techniques have successfully been applied to large source code corpora, yielding a variety of new software development tools, such as tools for code suggestion, improving readability, and API migration. A major issue with these techniques is that code introduces new vocabulary at a far higher rate than natural language, as new identifier names proliferate. Both large vocabularies and out-of-vocabulary issues severely affect Neural Language Models (NLMs) of source code, degrading their performance and rendering them unable to scale. In this paper, we address this issue by: 1) studying how various modelling choices impact the resulting vocabulary on a large-scale corpus of 13,362 projects; 2) presenting an open vocabulary source code NLM that can scale to such a corpus, 100 times larger than in previous work; and 3) showing that such models outperform the state of the art on three distinct code corpora (Java, C, Python). To our knowledge, these are the largest NLMs for code that have been reported. All datasets, code, and trained models used in this work are publicly available.Keywords
All Related Versions
- Version 1, 2020-03-17, ArXiv
- Published version: , 1073.
This publication has 52 references indexed in Scilit:
- When Code Completion Fails: A Case Study on Real-World Completions2019
- NL2Type: Inferring JavaScript Function Types from Natural Language Information2019
- Natural Software Revisited2019
- On Learning Meaningful Code Changes Via Neural Machine Translation2019
- Leveraging Small Software Engineering Data Sets with Pre-Trained Neural Networks2019
- An empirical study of identifier splitting techniquesEmpirical Software Engineering, 2013
- LINSEN: An efficient approach to split identifiers and expand abbreviations2012
- Mining source code to automatically split identifiers for software analysis2009
- To camelcase or under_score2009
- The Porter stemming algorithm: then and nowProgram: electronic library and information systems, 2006