Code Prediction by Feeding Trees to Transformers
Authors:
- Seohyun KimSKSeohyun Kim
- Jinman ZhaoJZJinman Zhao
- Yuchi TianYTYuchi Tian
- Satish ChandraSCSatish Chandra
Preprint
Abstract
We advance the state-of-the-art in the accuracy of code prediction (next token prediction) used in autocomplete systems. First, we report that using the recently proposed Transformer architecture even out-of-the-box outperforms previous neural and non-neural systems for code prediction. We then show that by making the Transformer architecture aware of the syntactic structure of code, we further increase the margin by which a Transformer-based system outperforms previous systems. With this, it outperforms the accuracy of an RNN-based system (similar to Hellendoorn et al. 2018) by 18.3\%, the Deep3 system (Raychev et al 2016) by 14.1\%, and an adaptation of Code2Seq (Alon et al., 2018) for code prediction by 14.4\%. We present in the paper several ways of communicating the code structure to the Transformer, which is fundamentally built for processing sequence data. We provide a comprehensive experimental evaluation of our proposal, along with alternative design choices, on a standard Python dataset, as well as on a Facebook internal Python corpus. Our code and data preparation pipeline will be available in open source.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Modeling programs hierarchically with stack-augmented LSTMJournal of Systems and Software, 2020
- Pythia: AI-assisted Code Completion System2019
- Character-Level Language Modeling with Deeper Self-AttentionProceedings of the AAAI Conference on Artificial Intelligence, 2019
- code2vec: learning distributed representations of codeProceedings of the ACM on Programming Languages, 2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Probabilistic model for code with decision trees2016
- Learning programs from noisy dataACM SIGPLAN Notices, 2016
- Intelligent Code Completion with Bayesian NetworksACM Transactions on Software Engineering and Methodology, 2015
- Code completion with statistical language models2014
- A statistical semantic language model for source code2013