Identifying Context-Dependent Translations for Evaluation Set Production
Preprint
Abstract
A major impediment to the transition to context-aware machine translation is the absence of good evaluation metrics and test sets. Sentences that require context to be translated correctly are rare in test sets, reducing the utility of standard corpus-level metrics such as COMET or BLEU. On the other hand, datasets that annotate such sentences are also rare, small in scale, and available for only a few languages. To address this, we modernize, generalize, and extend previous annotation pipelines to produce CTXPRO, a tool that identifies subsets of parallel documents containing sentences that require context to correctly translate five phenomena: gender, formality, and animacy for pronouns, verb phrase ellipsis, and ambiguous noun inflections. The input to the pipeline is a set of hand-crafted, per-language, linguistically-informed rules that select contextual sentence pairs using coreference, part-of-speech, and morphological features provided by state-of-the-art tools. We apply this pipeline to seven languages pairs (EN into and out-of DE, ES, FR, IT, PL, PT, and RU) and two datasets (OpenSubtitles and WMT test sets), and validate its performance using both overlap with previous work and its ability to discriminate a contextual MT system from a sentence-based one. We release the CTXPRO pipeline and data as open source.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- When Does Translation Require Context? A Data-driven, Multilingual Exploration2023
- F-coref: Fast, Accurate and Easy to Use Coreference Resolution2022
- MT-GenEval: A Counterfactual and Contextual Dataset for Evaluating Gender Accuracy in Machine Translation2022
- G-Transformer for Document-Level Machine Translation2021
- COMET: A Neural Framework for MT Evaluation2020
- SimAlign: High Quality Word Alignments Without Parallel Training Data Using Static and Contextualized Embeddings2020
- Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation2019
- When a Good Translation is Wrong in Context: Context-Aware Machine Translation Improves on Deixis, Ellipsis, and Lexical Cohesion2019
- Document-Level Neural Machine Translation with Hierarchical Attention Networks2018
- Evaluating Discourse Phenomena in Neural Machine Translation2018