Post-Processing of Word Representations via Variance Normalization and Dynamic Embedding
Authors:
- 1 July 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 718-723
Abstract
Language processing becomes more and more important in multimedia processing. Although embedded vector representations of words offer impressive performance on many natural language processing (NLP) applications, the information of ordered input sequences is lost to some extent if only context-based samples are used in the training. For further performance improvement, two new post-processing techniques, called post-processing via variance normalization (PVN) and post-processing via dynamic embedding (PDE), are proposed in this work. The PVN method normalizes the variance of principal components of word vectors, while the PDE method learns orthogonal latent variables from ordered input sequences. The PVN and the PDE methods can be integrated to achieve better performance. We apply these post-processing techniques to several popular word embedding methods to yield their post-processed representations. Extensive experiments are conducted to demonstrate the effectiveness of the proposed post-processing techniques.Keywords
This publication has 13 references indexed in Scilit:
- Evaluating word embedding models: methods and experimental resultsAPSIPA Transactions on Signal and Information Processing, 2019
- A novel dynamic PCA algorithm for dynamic data modeling and process monitoringJournal of Process Control, 2018
- Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context2018
- Deep Contextualized Word Representations2018
- Visualizing and Understanding Neural Machine Translation2017
- Dict2vec : Learning Word Embeddings using Lexical Dictionaries2017
- Learning principled bilingual mappings of word embeddings while preserving monolingual invariance2016
- Problems With Evaluation of Word Embeddings Using Word Similarity Tasks2016
- On the existence of obstinate results in vector space models2010
- On-line batch process monitoring using dynamic PCA and dynamic PLS modelsChemical Engineering Science, 2002