Essential Dimensions of Latent Semantic Indexing (LSI)
Author:
- 1 January 2007
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 73
Abstract
Latent semantic indexing (LSI) is commonly used to match queries to documents in information retrieval applications. LSI has been shown to improve retrieval performance for some, but not all, collections, when compared to traditional vector space retrieval. In this paper, we first develop a model for understanding which values in the reduced dimensional space contain the term relationship (latent semantic) information. We then test this model by developing a modified version of LSI that captures this information, essential dimensions of LSI (EDLSI). EDLSI significantly improves retrieval performance on corpora that previously did not benefit from LSI, and offers improved runtime performance when compared with traditional LSI. Traditional LSI requires the use of a dimensionality reduction parameter which must be tuned for each collection. Applying our model, we have also shown that a small, fixed dimensionality reduction parameter (k=10) can be used to capture the term relationship information in a corpusKeywords
This publication has 16 references indexed in Scilit:
- A framework for understanding Latent Semantic Indexing (LSI) performanceInformation Processing & Management, 2006
- Principal Component Analysis for Information Retrieval2005
- K-means clustering via principal component analysis2004
- Using LSI for text classification in the presence of background text2001
- Random projection in dimensionality reduction2001
- Matrices, Vector Spaces, and Information RetrievalSIAM Review, 1999
- An explanation of the effectiveness of latent semantic indexing by means of a Bayesian regression modelInformation Processing & Management, 1996
- Using Linear Algebra for Intelligent Information RetrievalSIAM Review, 1995
- OHSUMED: An Interactive Retrieval Evaluation and New Large Test Collection for Research1994
- Indexing by latent semantic analysisJournal of the American Society for Information Science, 1990