Empirical study of transformers for source code
Open Access
- 18 August 2021
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 703-715
Abstract
Initially developed for natural language processing (NLP), Transformers are now widely used for source code processing, due to the format similarity between source code and text. In contrast to natural language, source code is strictly structured, i.e., it follows the syntax of the programming language. Several recent works develop Transformer modifications for capturing syntactic information in source code. The drawback of these works is that they do not compare to each other and consider different tasks. In this work, we conduct a thorough empirical study of the capabilities of Transformers to utilize syntactic information in different tasks. We consider three tasks (code completion, function naming and bug fixing) and re-implement different syntax-capturing modifications in a unified framework. We show that Transformers are able to make meaningful predictions based purely on syntactic information and underline the best practices of taking the syntactic information into account for improving the performance of the model.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Code Completion with Neural Attention and Pointer Networks2018
- Deep code comment generation2018
- Compilation error repair2018
- Self-Attention with Relative Position Representations2018
- Probabilistic model for code with decision treesACM SIGPLAN Notices, 2016
- Probabilistic model for code with decision trees2016
- Learning programs from noisy data2016
- Summarizing Source Code using a Neural Attention Model2016
- Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks2015
- Long Short-Term MemoryNeural Computation, 1997