Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks
Authors:
- Nadezhda ChirkovaNCNadezhda Chirkova
- Vassilina NikoulinaVNVassilina Nikoulina
Preprint
Abstract
Zero-shot cross-lingual generation implies finetuning of the multilingual pretrained language model on a generation task in one language and then using it to make predictions for this task in other languages. Previous works notice a frequent problem of generation in a wrong language and propose approaches to address it, usually using mT5 as a backbone model. In this work we compare various approaches proposed from the literature in unified settings, also including alternative backbone models, namely mBART and NLLB-200. We first underline the importance of tuning learning rate used for finetuning, which helps to substantially alleviate the problem of generation in the wrong language. Then, we show that with careful learning rate tuning, the simple full finetuning of the model acts as a very strong baseline and alternative approaches bring only marginal improvements. Finally, we find that mBART performs similarly to mT5 of the same size, and NLLB-200 can be competitive in some cases. Our final models reach the performance of the approach based on data translation which is usually considered as an upper baseline for zero-shot cross-lingual generation.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- The Power of Scale for Parameter-Efficient Prompt Tuning2021
- A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters2021
- ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation2021
- XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages2021
- Multilingual Translation from Denoising Pre-Training2021
- Multilingual Denoising Pre-training for Neural Machine TranslationTransactions of the Association for Computational Linguistics, 2020
- From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers2020
- MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer2020
- Unsupervised Cross-lingual Representation Learning at Scale2020
- On the Cross-lingual Transferability of Monolingual Representations2020