Leveraging GPT-4 for Automatic Translation Post-Editing
Authors:
- Vikas RaunakVRVikas Raunak
- Amr SharafASAmr Sharaf
- Yiren WangYWYiren Wang
- Hany Hassan AwadallahHAHany Hassan Awadallah
- Arul MenezesAMArul Menezes
Preprint
Abstract
While Neural Machine Translation (NMT) represents the leading approach to Machine Translation (MT), the outputs of NMT models still require translation post-editing to rectify errors and enhance quality under critical settings. In this work, we formalize the task of direct translation post-editing with Large Language Models (LLMs) and explore the use of GPT-4 to automatically post-edit NMT outputs across several language pairs. Our results demonstrate that GPT-4 is adept at translation post-editing, producing meaningful and trustworthy edits to translations that help improve its general quality as well as remove different classes of major errors in translations. In particular, human evaluations on assessing edit trustworthiness show that GPT-4 exhibits a large improvement over the prior state-of-the-art LLM. Notably, we improve upon state-of-the-art performance on WMT-22 English-Chinese, English-German, Chinese-English and German-English language pairs using GPT-4 based post-editing, as evaluated by state-of-the-art MT quality metrics. However, we also show that GPT-4 could produce hallucinated edits, thereby urging caution in its use as an expert translation post-editor.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- How to Design Translation Prompts for ChatGPT: An Empirical Study2024
- Understanding and Detecting Hallucinations in Neural Machine Translation via Model IntrospectionTransactions of the Association for Computational Linguistics, 2023
- Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine TranslationTransactions of the Association for Computational Linguistics, 2021
- Testing Machine Translation via Referential Transparency2021
- A review of the state-of-the-art in automatic post-editingMachine Translation, 2020
- Machine translation testing via pathological invariance2020
- A roadmap to neural automatic post-editing: an empirical approachMachine Translation, 2020
- Structure-invariant testing for machine translation2020
- Automatic testing and improvement of machine translation2020
- COMET: A Neural Framework for MT Evaluation2020