Audio-Based Crowd-Sourced Evaluation of Machine Translation Quality
Preprint
Abstract
Machine Translation (MT) has achieved remarkable performance, with growing interest in speech translation and multimodal approaches. However, despite these advancements, MT quality assessment remains largely text centric, typically relying on human experts who read and compare texts. Since many real-world MT applications (e.g Google Translate Voice Mode, iFLYTEK Translator) involve translation being spoken rather printed or read, a more natural way to assess translation quality would be through speech as opposed text-only evaluations. This study compares text-only and audio-based evaluations of 10 MT systems from the WMT General MT Shared Task, using crowd-sourced judgments collected via Amazon Mechanical Turk. We additionally, performed statistical significance testing and self-replication experiments to test reliability and consistency of audio-based approach. Crowd-sourced assessments based on audio yield rankings largely consistent with text only evaluations but, in some cases, identify significant differences between translation systems. We attribute this to speech richer, more natural modality and propose incorporating speech-based assessments into future MT evaluation frameworks.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Evaluating the IWSLT2023 Speech Translation Tasks: Human Annotations, Automatic Metrics, and Segmentation2024
- Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation2024
- Findings of the 2023 Conference on Machine Translation (WMT23): LLMs Are Here but Not Quite There Yet2023
- Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine TranslationTransactions of the Association for Computational Linguistics, 2021
- Choice of Voices2020
- Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation2018
- Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation2018
- Can machine translation systems be evaluated by the crowd aloneNatural Language Engineering, 2015
- Task Requirements and Media Choice in Collaborative WritingHuman–Computer Interaction, 1992