Evaluating Code Summarization Techniques: A New Metric and an Empirical Characterization
Open Access
- 12 April 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-13
Abstract
Several code summarization techniques have been proposed in the literature to automatically document a code snippet or a function. Ideally, software developers should be involved in assessing the quality of the generated summaries. However, in most cases, researchers rely on automatic evaluation metrics such as BLEU, ROUGE, and METEOR. These metrics are all based on the same assumption: The higher the textual similarity between the generated summary and a reference summary written by developers, the higher its quality. However, there are two reasons for which this assumption falls short: (i) reference summaries, e.g., code comments collected by mining software repositories, may be of low quality or even outdated; (ii) generated summaries, while using a different wording than a reference one, could be semantically equivalent to it, thus still being suitable to document the code snippet. In this paper, we perform a thorough empirical investigation on the complementarity of different types of metrics in capturing the quality of a generated summary. Also, we propose to address the limitations of existing metrics by considering a new dimension, capturing the extent to which the generated summary aligns with the semantics of the documented code snippet, independently from the reference summary. To this end, we present a new metric based on contrastive learning to capture said aspect. We empirically show that the inclusion of this novel dimension enables a more effective representation of developers' evaluations regarding the quality of automatically generated summaries.Keywords
All Related Versions
Funding Information
- European Research Council (851720)
- Italian Ministry of Research (2020W3A5FY)
This publication has 60 references indexed in Scilit:
- FacTeR-Check: Semi-automated fact-checking through semantic similarity and natural language inferenceKnowledge-Based Systems, 2022
- Deep Code-Comment Understanding and AssessmentIEEE Access, 2019
- Automatically detecting the scopes of source code commentsJournal of Systems and Software, 2019
- Deep code comment generation with hybrid lexical and syntactical informationEmpirical Software Engineering, 2019
- Automatic Quality Assessment of Source Code Comments: The JavadocMiner2010
- Analyzing the co-evolution of comments and source codeSoftware Quality Journal, 2009
- Dictionary of Bioinformatics and Computational Biology2004
- Time Series Analysis2002
- Robust Estimation of a Location Parameter1992
- Program readability: procedures versus commentsIEEE Transactions on Software Engineering, 1988