Regularized Latent Semantic Indexing
Authors:
Open Access
- 1 January 2013
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 31 (1) , 1-44
Abstract
Topic modeling provides a powerful way to analyze the content of a collection of documents. It has become a popular tool in many research areas, such as text mining, information retrieval, natural language processing, and other related fields. In real-world applications, however, the usefulness of topic modeling is limited due to scalability issues. Scaling to larger document collections via parallelization is an active area of research, but most solutions require drastic steps, such as vastly reducing input vocabulary. In this article we introduce Regularized Latent Semantic Indexing (RLSI)---including a batch version and an online version, referred to as batch RLSI and online RLSI, respectively---to scale up topic modeling. Batch RLSI and online RLSI are as effective as existing topic modeling techniques and can scale to larger datasets without reducing input vocabulary. Moreover, online RLSI can be applied to stream data and can capture the dynamic evolution of topics. Both versions of RLSI formalize topic modeling as a problem of minimizing a quadratic loss function regularized by ℓ1 and/or ℓ2 norm. This formulation allows the learning process to be decomposed into multiple suboptimization problems which can be optimized in parallel, for example, via MapReduce. We particularly propose adopting ℓ1 norm on topics and ℓ2 norm on document representations to create a model with compact and readable topics and which is useful for retrieval. In learning, batch RLSI processes all the documents in the collection as a whole, while online RLSI processes the documents in the collection one by one. We also prove the convergence of the learning of online RLSI. Relevance ranking experiments on three TREC datasets show that batch RLSI and online RLSI perform better than LSI, PLSI, LDA, and NMF, and the improvements are sometimes statistically significant. Experiments on a Web dataset containing about 1.6 million documents and 7 million terms, demonstrate a similar boost in performance.Keywords
This publication has 36 references indexed in Scilit:
- PLDA+ACM Transactions on Intelligent Systems and Technology, 2011
- Latent semantic indexing (LSI) fails for TREC collectionsACM SIGKDD Explorations Newsletter, 2011
- A probabilistic model for Latent Semantic IndexingJournal of the American Society for Information Science and Technology, 2005
- Optimization of Collective Communication Operations in MPICHThe International Journal of High Performance Computing Applications, 2005
- Least angle regressionThe Annals of Statistics, 2004
- A new approach to variable selection in least squares problemsIMA Journal of Numerical Analysis, 2000
- Penalized Regressions: The Bridge versus the LassoJournal of Computational and Graphical Statistics, 1998
- Sparse coding with an overcomplete basis set: A strategy employed by V1?Vision Research, 1997
- Indexing by latent semantic analysisJournal of the American Society for Information Science, 1990
- A vector space model for automatic indexingCommunications of the ACM, 1975