PBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embedding
Preprint
Abstract
We look into the task of \emph{generalizing} word embeddings: given a set of pre-trained word vectors over a finite vocabulary, the goal is to predict embedding vectors for out-of-vocabulary words, \emph{without} extra contextual information. We rely solely on the spellings of words and propose a model, along with an efficient algorithm, that simultaneously models subword segmentation and computes subword-based compositional word embedding. We call the model probabilistic bag-of-subwords (PBoS), as it applies bag-of-subwords for all possible segmentations based on their likelihood. Inspections and affix prediction experiment show that PBoS is able to produce meaningful subword segmentations and subword rankings without any source of explicit morphological knowledge. Word similarity and POS tagging experiments show clear advantages of PBoS over previous subword-level models in the quality of generated word embeddings across languages.Keywords
All Related Versions
This publication has 34 references indexed in Scilit:
- Wikipedia2Vec: An Efficient Toolkit for Learning and Visualizing the Embeddings of Words and Entities from Wikipedia2020
- Learning Semantic Representations for Novel Words: Leveraging Both Form and ContextProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Enriching Word Vectors with Subword InformationTransactions of the Association for Computational Linguistics, 2017
- Segmentation-Free Word Embedding for Unsegmented Languages2017
- Mimicking Word Embeddings using Subword RNNs2017
- Charagram: Embedding Words and Sentences via Character n-grams2016
- Morphological Priors for Probabilistic Neural Word Embeddings2016
- KNETACM Transactions on Information Systems, 2015
- The indeterminacy of word segmentation and the nature of morphology and syntaxFolia Linguistica, 2011
- Placing search in context2001