What Do Language Representations Really Represent?
Open Access
- 1 June 2019
- journal article
- editorial
- Published by MIT Press in Computational Linguistics
- Vol. 45 (2) , 381-389
Abstract
A neural language model trained on a text corpus can be used to induce distributed representations of words, such that similar words end up with similar representations. If the corpus is multilingual, the same model can be used to learn distributed representations of languages, such that similar languages end up with similar representations. We show that this holds even when the multilingual corpus has been translated into English, by picking up the faint signal left by the source languages. However, just as it is a thorny problem to separate semantic from syntactic similarity in word representations, it is not obvious what type of similarity is captured by language representations. We investigate correlations and causal relationships between language representations learned from translations on one hand, and genetic, geographical, and several levels of structural similarity between languages on the other. Of these, structural similarity is found to correlate most strongly with language representation similarity, whereas genetic relationships—a convenient benchmark used for evaluation in previous work—appears to be a confounding factor. Apart from implications about translation effects, we see this more generally as a case where NLP and linguistic typology can interact and benefit one another.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Language Typology and Language Universals2021
- On the Relation between Linguistic Typology and (Limitations of) Multilingual Language Modeling2018
- From Phonology to Syntax: Unsupervised Linguistic Typology at Different Levels with Language Embeddings2018
- Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot TranslationTransactions of the Association for Computational Linguistics, 2017
- Continuous multilinguality with language vectors2017
- Probabilistic Typology: Deep Generative Models of Vowel Inventories2017
- Learning Language Representations for Typology Prediction2017
- Many Languages, One ParserTransactions of the Association for Computational Linguistics, 2016
- Word Order Typology through Multilingual Word Alignment2015
- Evolved structure of language shows lineage-specific trends in word-order universalsNature, 2011