SemSyn: Semantic-Syntactic Similarity Based Automatic Machine Translation Evaluation Metric
Top Cited Papers
- 2 April 2023
- journal article
- research article
- Published by Taylor & Francis in IETE Journal of Research
- Vol. 70 (4) , 3823-3834
Abstract
Machine translation evaluation is difficult and challenging for natural languages because different languages behave differently for the same dataset. Lexical-based metrics have been poorly represented semantic relationships and impose strict identity matching. However, translation and assessment become difficult for target morphologically rich languages with relatively free word order. Most of the standard evaluation metrics consider word order but do not effectively consider sentence structure. In this paper, we propose a novel machine translation evaluation metric SemSyn which incorporates both semantic and syntactic similarity. We incorporate the term frequency-inverse document frequency with the earth mover’s distance and word embedding to cover the semantic similarity. The part of speech and dependency parsing tags assist in covering syntactic similarity in the sentence structure. Part of speech and dependency parsing tags are extracted from universal dependencies and trained on the SpaCy library. Experimental results show that SemSyn has a higher correlation with human judgment than other evaluation metrics for morphologically rich language and other languages.Keywords
This publication has 13 references indexed in Scilit:
- How to evaluate machine translation: A review of automated and human metricsNatural Language Engineering, 2019
- The Hindi/Urdu Treebank Project2017
- Bag of Tricks for Efficient Text Classification2017
- UHH Submission to the WMT17 Quality Estimation Shared Task2017
- Evaluating MT systems with BEERThe Prague Bulletin of Mathematical Linguistics, 2015
- Using Discourse Structure Improves Machine Translation Evaluation2014
- Robust hand gesture recognition based on finger-earth mover's distance with a commodity depth camera2011
- ATEC: automatic evaluation of machine translation via word choice and word orderMachine Translation, 2009
- Fluency, adequacy, or HTER?2009
- Automatic evaluation of machine translation quality using n-gram co-occurrence statistics2002