Language model-based document clustering using random walks
Author:
Open Access
- 1 January 2006
- conference paper
- Published by Association for Computational Linguistics (ACL)
- p. 479-486
Abstract
We propose a new document vector representation specifically designed for the document clustering task. Instead of the traditional termbased vectors, a document is represented as an-dimensional vector, where is the number of documents in the cluster. The value at each dimension of the vector is closely related to the generation probability based on the language model of the corresponding document. Inspired by the recent graph-based NLP methods, we reinforce the generation probabilities by iterating random walks on the underlying graph representation. Experiments with k-means and hierarchical clustering algorithms show significant improvements over the alternative ¡¢ £¤¥¢ vector representation.Keywords
This publication has 6 references indexed in Scilit:
- LexRank: Graph-based Lexical Centrality as Salience in Text SummarizationJournal of Artificial Intelligence Research, 2004
- A study of smoothing methods for language models applied to information retrievalACM Transactions on Information Systems, 2004
- The use of bigrams to enhance text categorizationInformation Processing & Management, 2002
- Document clustering using word clusters via the information bottleneck method2000
- Collective dynamics of ‘small-world’ networksNature, 1998
- Algorithms for the Assignment and Transportation ProblemsJournal of the Society for Industrial and Applied Mathematics, 1957