Matrix Factorization using Window Sampling and Negative Sampling for Improved Word Representations
Preprint
Abstract
In this paper, we propose LexVec, a new method for generating distributed word representations that uses low-rank, weighted factorization of the Positive Point-wise Mutual Information matrix via stochastic gradient descent, employing a weighting scheme that assigns heavier penalties for errors on frequent co-occurrences while still accounting for negative co-occurrence. Evaluation on word similarity and analogy tasks shows that LexVec matches and often outperforms state-of-the-art methods on many of these tasks.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- SimLex-999: Evaluating Semantic Models With (Genuine) Similarity EstimationComputational Linguistics, 2015
- Improving Distributional Similarity with Lessons Learned from Word EmbeddingsTransactions of the Association for Computational Linguistics, 2015
- Linguistic Regularities in Sparse and Explicit Word Representations2014
- Extracting semantic representations from word co-occurrence statistics: A computational studyBehavior Research Methods, 2007
- Contextual correlates of semantic similarityLanguage and Cognitive Processes, 1991
- Contextual correlates of synonymyCommunications of the ACM, 1965