Finding Universal Grammatical Relations in Multilingual BERT
Preprint
Abstract
Recent work has found evidence that Multilingual BERT (mBERT), a transformer-based multilingual masked language model, is capable of zero-shot cross-lingual transfer, suggesting that some aspects of its representations are shared cross-lingually. To better understand this overlap, we extend recent work on finding syntactic trees in neural networks' internal representations to the multilingual setting. We show that subspaces of mBERT representations recover syntactic tree distances in languages other than English, and that these subspaces are approximately shared across languages. Motivated by these results, we present an unsupervised analysis method that provides evidence mBERT learns representations of syntactic dependency labels, in the form of clusters which largely agree with the Universal Dependencies taxonomy. This evidence suggests that even without explicit supervision, multilingual masked language models learn certain linguistic universals.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Asking without Telling: Exploring Latent Ontologies in Contextual Representations2020
- Emerging Cross-lingual Structure in Pretrained Language Models2020
- Unsupervised Cross-lingual Representation Learning at Scale2020
- Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing2019
- Investigating Multilingual NMT Representations at Scale2019
- How Multilingual is Multilingual BERT?2019
- Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing2019
- CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies2018
- What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties2018
- How to Use t-SNE Effectively2016