Counterfactually Probing Language Identity in Multilingual Models
Preprint
Abstract
Techniques in causal analysis of language models illuminate how linguistic information is organized in LLMs. We use one such technique, AlterRep, a method of counterfactual probing, to explore the internal structure of multilingual models (mBERT and XLM-R). We train a linear classifier on a binary language identity task, to classify tokens between Language X and Language Y. Applying a counterfactual probing procedure, we use the classifier weights to project the embeddings into the null space and push the resulting embeddings either in the direction of Language X or Language Y. Then we evaluate on a masked language modeling task. We find that, given a template in Language X, pushing towards Language Y systematically increases the probability of Language Y words, above and beyond a third-party control language. But it does not specifically push the model towards translation-equivalent words in Language Y. Pushing towards Language X (the same direction as the template) has a minimal effect, but somewhat degrades these models. Overall, we take these results as further evidence of the rich structure of massive multilingual language models, which include both a language-specific and language-general component. And we show that counterfactual probing can be fruitfully applied to multilingual models.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Amnesic Probing: Behavioral Explanation with Amnesic CounterfactualsTransactions of the Association for Computational Linguistics, 2021
- From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers2020
- On the Language Neutrality of Pre-trained Multilingual Representations2020
- It’s not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT2020
- Probing Pretrained Language Models for Lexical Semantics2020
- Emerging Cross-lingual Structure in Pretrained Language Models2020
- Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection2020
- Unsupervised Cross-lingual Representation Learning at Scale2020
- Information-Theoretic Probing for Linguistic Structure2020
- Finding Universal Grammatical Relations in Multilingual BERT2020