Knowledge-Prompted Estimator: A Novel Approach to Explainable Machine Translation Assessment
Preprint
Abstract
Cross-lingual Machine Translation (MT) quality estimation plays a crucial role in evaluating translation performance. GEMBA, the first MT quality assessment metric based on Large Language Models (LLMs), employs one-step prompting to achieve state-of-the-art (SOTA) in system-level MT quality estimation; however, it lacks segment-level analysis. In contrast, Chain-of-Thought (CoT) prompting outperforms one-step prompting by offering improved reasoning and explainability. In this paper, we introduce Knowledge-Prompted Estimator (KPE), a CoT prompting method that combines three one-step prompting techniques, including perplexity, token-level similarity, and sentence-level similarity. This method attains enhanced performance for segment-level estimation compared with previous deep learning models and one-step prompting approaches. Furthermore, supplementary experiments on word-level visualized alignment demonstrate that our KPE method significantly improves token alignment compared with earlier models and provides better interpretability for MT quality estimation. Code will be released upon publication.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- Implicit Cross-Lingual Word Embedding Alignment for Reference-Free Machine Translation EvaluationIEEE Access, 2023
- Prompting PaLM for Translation: Assessing Strategies and Performance2023
- On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluation2020
- Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and BeyondTransactions of the Association for Computational Linguistics, 2019