The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer
Preprint
Abstract
Large multilingual language models such as mBERT or XLM-R enable zero-shot cross-lingual transfer in various IR and NLP tasks. Cao et al. (2020) proposed a data- and compute-efficient method for cross-lingual adjustment of mBERT that uses a small parallel corpus to make embeddings of related words across languages similar to each other. They showed it to be effective in NLI for five European languages. In contrast we experiment with a typologically diverse set of languages (Spanish, Russian, Vietnamese, and Hindi) and extend their original implementations to new tasks (XSR, NER, and QA) and an additional training regime (continual learning). Our study reproduced gains in NLI for four languages, showed improved NER, XSR, and cross-lingual QA results in three languages (though some cross-lingual QA gains were not statistically significant), while mono-lingual QA performance never improved and sometimes degraded. Analysis of distances between contextualized embeddings of related and unrelated words (across languages) showed that fine-tuning leads to "forgetting" some of the cross-lingual alignment information. Based on this observation, we further improved NLI performance using continual learning.Keywords
All Related Versions
- Version 1, 2022-04-13, ArXiv
- Version 2, 2023-10-31, ArXiv
- Published version: , 51.
This publication has 15 references indexed in Scilit:
- A Primer on Pretrained Multilingual Language ModelsACM Computing Surveys, 2025
- QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading ComprehensionACM Computing Surveys, 2023
- Can BERT Dig It? Named Entity Recognition for Information Retrieval in the Archaeology DomainJournal on Computing and Cultural Heritage, 2022
- Evaluating Multilingual Text Encoders for Unsupervised Cross-Lingual Retrieval2021
- HPC Resources of the Higher School of EconomicsJournal of Physics: Conference Series, 2021
- Building Large-Scale Deep Learning System for Entity Recognition in E-Commerce Search2019
- Document Retrieval Model Through Semantic Linking2017
- Catastrophic Forgetting, Rehearsal and PseudorehearsalConnection Science, 1995
- Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions.Psychological Review, 1990
- Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem1989