Prompting Large Language Model for Machine Translation: A Case Study
Preprint
Abstract
Research on prompting has shown excellent performance with little or even no supervised training across many tasks. However, prompting for machine translation is still under-explored in the literature. We fill this gap by offering a systematic study on prompting strategies for translation, examining various factors for prompt template and demonstration example selection. We further explore the use of monolingual data and the feasibility of cross-lingual, cross-domain, and sentence-to-document transfer learning in prompting. Extensive experiments with GLM-130B (Zeng et al., 2022) as the testbed show that 1) the number and the quality of prompt examples matter, where using suboptimal examples degenerates translation; 2) several features of prompt examples, such as semantic similarity, show significant Spearman correlation with their prompting performance; yet, none of the correlations are strong enough; 3) using pseudo parallel prompt examples constructed from monolingual data via zero-shot prompting could improve translation; and 4) improved performance is achievable by transferring knowledge from prompt examples selected in other settings. We finally provide an analysis on the model outputs and discuss several problems that prompting still suffers from.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels2022
- MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators2022
- What Makes Good In-Context Examples for GPT-3?2022
- TransPrompt: Towards an Automatic Transferable Prompting Framework for Few-shot Text Classification2021
- WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia2021
- Making Pre-trained Language Models Better Few-shot Learners2021
- mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer2021
- COMET: A Neural Framework for MT Evaluation2020
- Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation2020
- Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot TranslationTransactions of the Association for Computational Linguistics, 2017