Towards Summarizing Code Snippets Using Pre-Trained Transformers
Open Access
- 15 April 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Abstract
When comprehending code, a helping hand may come from the natural language comments documenting it that, unfortunately, are not always there. To support developers in such a scenario, several techniques have been presented to automatically generate natural language summaries for a given code. Most recent approaches exploit deep learning (DL) to automatically document classes or functions, while little effort has been devoted to more fine-grained documentation (e.g., documenting code snippets or even a single statement). Such a design choice is dictated by the availability of training data: For example, in the case of Java, it is easy to create datasets composed of pairs <method, javadoc> that can be fed to DL models to teach them how to summarize a method. Such a comment-to-code linking is instead non-trivial when it comes to inner comments documenting a few statements. In this work, we take all the steps needed to train a DL model to automatically document code snippets. First, we manually built a dataset featuring 6.6k comments that have been (i) classified based on their type (e.g., code summary, TODO), and (ii) linked to the code statements they document. Second, we used such a dataset to train a multi-task DL model taking as input a comment and being able to (i) classify whether it represents a "code summary" or not, and (ii) link it to the code statements it documents. Our model identifies code summaries with 84% accuracy and is able to link them to the documented lines of code with recall and precision higher than 80%. Third, we run this model on 10k projects, identifying and linking code summaries to the documented code. This unlocked the possibility of building a large-scale dataset of documented code snippets that have then been used to train a new DL model able to automatically document code snippets. A comparison with state-of-the-art baselines shows the superiority of the proposed approach, which however, is still far from representing an accurate solution for snippet summarization.Keywords
All Related Versions
Funding Information
- European Research Council (ERC) under the European Union?s Horizon 2020 research and innovation programme (851720)
This publication has 52 references indexed in Scilit:
- Towards automatically generating block comments for code snippetsInformation and Software Technology, 2020
- Automatically detecting the scopes of source code commentsJournal of Systems and Software, 2019
- Deep code comment generation with hybrid lexical and syntactical informationEmpirical Software Engineering, 2019
- CodeAttention: translating source code to comments by exploiting the code constructsFrontiers of Computer Science, 2018
- How do Developers Document Database Usages in Source Code? (N)2015
- An empirical study of the textual similarity between source code and source code summariesEmpirical Software Engineering, 2014
- AutoComment: Mining question and answer sites for automatic comment generation2013
- Analyzing the co-evolution of comments and source codeSoftware Quality Journal, 2009
- Dictionary of Bioinformatics and Computational Biology2004
- Note on the Sampling Error of the Difference Between Correlated Proportions or PercentagesPsychometrika, 1947