Zero-shot cross-lingual transfer in instruction tuning of large language models
Authors:
- Nadezhda ChirkovaNCNadezhda Chirkova
- Vassilina NikoulinaVNVassilina Nikoulina
Preprint
Abstract
Instruction tuning (IT) is widely used to teach pretrained large language models (LLMs) to follow arbitrary instructions, but is under-studied in multilingual settings. In this work, we conduct a systematic study of zero-shot cross-lingual transfer in IT, when an LLM is instruction-tuned on English-only data and then tested on user prompts in other languages. We advocate for the importance of evaluating various aspects of model responses in multilingual instruction following and investigate the influence of different model configuration choices. We find that cross-lingual transfer does happen successfully in IT even if all stages of model training are English-centric, but only if multiliguality is taken into account in hyperparameter tuning and with large enough IT data. English-trained LLMs are capable of generating correct-language, comprehensive and helpful responses in other languages, but suffer from low factuality and may occasionally have fluency errors.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuning2024
- Empowering cross-lingual abilities of instruction-tuned large language models by translation-following demonstrations2024
- Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation2022
- Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models2022
- mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer2021
- MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer2020
- Unsupervised Cross-lingual Representation Learning at Scale2020
- On the Cross-lingual Transferability of Monolingual Representations2020
- Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT2019
- How Multilingual is Multilingual BERT?2019