Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fast
Preprint
Abstract
Existing research has shown that a multilingual pre-trained language model fine-tuned with one (source) language also performs well on downstream tasks for non-source languages, even though no fine-tuning is done on these languages. However, there is a clear gap between the performance of the source language and that of the non-source languages. This paper analyzes the fine-tuning process, discovers when the performance gap changes and identifies which network weights affect the overall performance most. Additionally, the paper seeks to answer to what extent the gap can be reduced by reducing forgetting. Based on the analysis results, a method named Fine-tuning slow and fast with four training policies is proposed to address these issues. Experimental results show the proposed method outperforms baselines by a clear margin.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better2022
- Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning2021
- ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora2021
- Consistency Regularization for Cross-Lingual Fine-Tuning2021
- On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation2021
- InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training2021
- On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment2020
- Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting2020
- Unsupervised Cross-lingual Representation Learning at Scale2020
- MLQA: Evaluating Cross-lingual Extractive Question Answering2020