Semantic similarity metrics for evaluating source code summarization
Authors:
Top Cited Papers
Open Access
- 16 May 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 36-47
Abstract
Source code summarization involves creating brief descriptions of source code in natural language. These descriptions are a key component of software documentation such as JavaDocs. Automatic code summarization is a prized target of software engineering research, due to the high value summaries have to programmers and the simultaneously high cost of writing and maintaining documentation by hand. Current work is almost all based on machine models trained via big data input. Large datasets of examples of code and summaries of that code are used to train an e.g. encoder-decoder neural model. Then the output predictions of the model are evaluated against a set of reference summaries. The input is code not seen by the model, and the prediction is compared to a reference. The means by which a prediction is compared to a reference is essentially word overlap, calculated via a metric such as BLEU or ROUGE. The problem with using word overlap is that not all words in a sentence have the same importance, and many words have synonyms. The result is that calculated similarity may not match the perceived similarity by human readers. In this paper, we conduct an experiment to measure the degree to which various word overlap metrics correlate to human-rated similarity of predicted and reference summaries. We evaluate alternatives based on current work in semantic similarity metrics and propose recommendations for evaluation of source code summarization.Keywords
All Related Versions
Funding Information
- NSF (CCF-2100035)
This publication has 22 references indexed in Scilit:
- Automatic Source Code Summarization of Context for Java MethodsIEEE Transactions on Software Engineering, 2015
- Automatic Summarization of Bug ReportsIEEE Transactions on Software Engineering, 2014
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- Meta-analysis of Cohen’s kappaHealth Services and Outcomes Research Methodology, 2011
- Agreement and Information in the Reliability of CodingCommunication Methods and Measures, 2011
- Wanted: guidelines for reporting correlationsAdvances in Physiology Education, 2009
- Three Approaches to Qualitative Content AnalysisQualitative Health Research, 2005
- Evaluating Discourse and Dialogue Coding SchemesComputational Linguistics, 2005
- Rethinking validity and reliability in content analysisJournal of Applied Communication Research, 1999
- Term-weighting approaches in automatic text retrievalInformation Processing & Management, 1988