Intrinsic or Extrinsic Evaluation: An Overview of Word Embedding Evaluation
Authors:
- Yong ShiYSYong Shi
- Yuanchun ZhengYZYuanchun Zheng
- Kun GuoKGKun Guo
- Luyao ZhuLZLuyao Zhu
- Yi QuYQYi Qu
- 1 November 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1255-1262
Abstract
Compared with traditional methods, word em-bedding is an efficient language representation that can learn syntax and semantics by using neural networks. As the result, more and more promising experiments in natural language processing (NLP) get the state-of-the-art results by introducing word embedding. In principle, embedding representation learning embeds words to a low-dimensional vector space, there-fore vectors support initialization of NLP tasks such as text classification, sentiment analysis, language understanding, etc. However, polysemy is very common in many languages, which causes word ambiguation, further influences the accuracy of the system. Additionally, language models based on distributed hypotheses mostly focused on word properties rather than morphology were our primary focus. This leads to unreasonable performance in different evaluations. At the same time, word embedding learning and measuring are two vital components of word representation. In this paper, we overviewed many language models including single sense and multiple sense word embedding, and many evaluated approaches including intrinsic and extrinsic evaluation. We found that there are obvious gaps between vectors and manual annotations in word similarity evaluation, and language models that achieved good performance in intrinsic evaluations could not produce similar results in extrinsic evaluations. To the best of our knowledge, there is no universal language model and embedding learning method for most NLP task, and each evaluations also hidden natural defects compared to human knowledge. More evaluated datasets are also investigated such as datasets used in intrinsic and extrinsic evaluations. We believe that an improved evaluation dataset and a more rational evaluation method would benefit from this overview.Keywords
This publication has 27 references indexed in Scilit:
- SensEmbed: Learning Sense Embeddings for Word and Relational Similarity2015
- Don't count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors2014
- Glove: Global Vectors for Word Representation2014
- A Unified Model for Word Sense Representation and Disambiguation2014
- Lexicon-Based Methods for Sentiment AnalysisComputational Linguistics, 2011
- Distributional Memory: A General Framework for Corpus-Based SemanticsComputational Linguistics, 2010
- Modeling the Influence of Thematic Fit (and Other Constraints) in On-line Sentence ComprehensionJournal of Memory and Language, 1998
- An introduction to latent semantic analysisDiscourse Processes, 1998
- A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge.Psychological Review, 1997
- Contextual correlates of synonymyCommunications of the ACM, 1965