Code Prediction by Feeding Trees to Transformers
Preprint
Abstract
In this paper, we describe how to leverage \emph{Transformer}, a recent neural architecture for learning from sequential data (such as text), for code completion. As in the realm of natural language processing, Transformers surpass the prediction accuracy achievable by RNNs; we provide an experimental confirmation of this over a Python dataset. Furthermore, we show that the way to obtain even better accuracy from Transformers is to expose the syntactic structure of code, which is easily recovered by parsing, to the neural network. This works significantly better than presenting the code as a linear token sequence, which is how Transformers were originally intended to be used. To accomplish this, we propose a novel enhancement to the self-attention mechanism of the Transformer. We enable the mechanism to learn weights---that is, how much to focus on each preceding token in the input---not only on the basis of a token's value, but also on the basis of the spatial relationships, as in their positions in the abstract syntax tree, between each pair of tokens. We provide comprehensive experimental evaluation of our proposal, along with alternative design choices, on a standard Python dataset, as well as on a Python corpus internal to Facebook.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Modeling programs hierarchically with stack-augmented LSTMJournal of Systems and Software, 2020
- Character-Level Language Modeling with Deeper Self-AttentionProceedings of the AAAI Conference on Artificial Intelligence, 2019
- code2vec: learning distributed representations of codeProceedings of the ACM on Programming Languages, 2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Building Language Models for Text with Named Entities2018
- Probabilistic model for code with decision trees2016
- Learning programs from noisy dataACM SIGPLAN Notices, 2016
- Summarizing Source Code using a Neural Attention Model2016
- Code completion with statistical language models2014
- A statistical semantic language model for source code2013