Does BLEU Score Work for Code Migration?
Preprint
Abstract
Statistical machine translation (SMT) is a fast-growing sub-field of computational linguistics. Until now, the most popular automatic metric to measure the quality of SMT is BiLingual Evaluation Understudy (BLEU) score. Lately, SMT along with the BLEU metric has been applied to a Software Engineering task named code migration. (In)Validating the use of BLEU score could advance the research and development of SMT-based code migration tools. Unfortunately, there is no study to approve or disapprove the use of BLEU score for source code. In this paper, we conducted an empirical study on BLEU score to (in)validate its suitability for the code migration task due to its inability to reflect the semantics of source code. In our work, we use human judgment as the ground truth to measure the semantic correctness of the migrated code. Our empirical study demonstrates that BLEU does not reflect translation quality due to its weak correlation with the semantic correctness of translated code. We provided counter-examples to show that BLEU is ineffective in comparing the translation quality between SMT-based models. Due to BLEU's ineffectiveness for code migration task, we propose an alternative metric RUBY, which considers lexical, syntactical, and semantic representations of source code. We verified that RUBY achieves a higher correlation coefficient with the semantic correctness of migrated code, 0.775 in comparison with 0.583 of BLEU score. We also confirmed the effectiveness of RUBY in reflecting the changes in translation quality of SMT-based translation models. With its advantages, RUBY can be used to evaluate SMT-based code migration models.Keywords
All Related Versions
- Version 1, 2019-06-12, ArXiv
- Published version: , 165.
This publication has 10 references indexed in Scilit:
- Phrase-Based Statistical Translation of Programming Languages2014
- Wiley StatsRef: Statistics Reference Online2014
- Comparison and evaluation of code clone detection techniques and tools: A qualitative approachScience of Computer Programming, 2009
- Accurate and Efficient Structural Characteristic Feature Extraction for Clone Detection2009
- Comparison and Evaluation of Clone Detection ToolsIEEE Transactions on Software Engineering, 2007
- CP-Miner: finding copy-paste and related bugs in large-scale software codeIEEE Transactions on Software Engineering, 2006
- CCFinder: a multilinguistic token-based code clone detection system for large scale source codeIEEE Transactions on Software Engineering, 2002
- Parameterized Duplication in Strings: Algorithms and an Application to Software MaintenanceSIAM Journal on Computing, 1997
- Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography1987
- A distance measure between attributed relational graphs for pattern recognitionIEEE Transactions on Systems, Man, and Cybernetics, 1983