Word Embedding With Zipf’s Context
Open Access
- 20 November 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 7, 168934-168943
Abstract
Word embeddings generated by neural language models have achieved great success in many NLP tasks. However, neural language models may be difficult to train and time consuming. In this paper, we introduce a simpler but efficient word embedding method based on cooccurrence matrix factorization. Our method drastically reduces the dimensions of the cooccurrence matrix according to the famous Zipf's word frequency law. We observe that if the sampling times of a target word increase to a certain extent, the context of the target word will follow the Zipf's distribution. Enlightened by this, we propose a novel transformation for the cooccurrence matrix. The built cooccurrence matrix is then factorized by PCA. As PCA simply factorizes the cooccurrence matrix linearly and cannot capture the nonlinear relations of features, we construct an autoencoder to further transform the vectors. We compare our method with some well-known neural language models. Our method shows a comparable performance though it is much simpler than the neural language models.Keywords
Funding Information
- National Natural Science Foundation of China (61602508, 61866008)
This publication has 13 references indexed in Scilit:
- Bag of Tricks for Efficient Text Classification2017
- Glove: Global Vectors for Word Representation2014
- Word Embeddings through Hellinger PCA2014
- Extracting semantic representations from word co-occurrence statistics: A computational studyBehavior Research Methods, 2007
- Three new graphical models for statistical language modelling2007
- Placing search in contextACM Transactions on Information Systems, 2002
- Producing high-dimensional semantic spaces from lexical co-occurrenceBehavior Research Methods, Instruments, & Computers, 1996
- Contextual correlates of semantic similarityLanguage and Cognitive Processes, 1991
- Contextual correlates of synonymyCommunications of the ACM, 1965
- Distributional StructureWORD, 1954