Real-Time Top-R Topic Detection on Twitter with Topic Hijack Filtering
Open Access
- 10 August 2015
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 417-426
Abstract
Twitter is a \"what's-happening-right-now\" tool that enables interested parties to follow thoughts and commentary of individual users in nearly real-time. While it is a valuable source of information for real-time topic detection and tracking, Twitter data are not clean because of noisy messages and users, which significantly diminish the reliability of obtained results. In this paper, we integrate both the extraction of meaningful topics and the filtering of messages over the Twitter stream. We develop a streaming algorithm for a sequence of document-frequency tables; our algorithm enables real-time monitoring of the top-10 topics from approximately 25% of all Twitter messages, while automatically filtering noisy and meaningless topics. We apply our proposed streaming algorithm to the Japanese Twitter stream and successfully demonstrate that, compared with other online nonnegative matrix factorization methods, our framework both tracks real-world events with high accuracy in terms of the perplexity and simultaneously eliminates irrelevant topics.Keywords
This publication has 28 references indexed in Scilit:
- On-line learning parts-based representation via incremental orthogonal projective non-negative matrix factorizationSignal Processing, 2013
- Learning evolving and emerging topics in social media2012
- Efficient Document Clustering via Online Nonnegative Matrix Factorizations2011
- Spam Filtering in Twitter Using Sender-Receiver Relationship2011
- Power-Law Distributions in Empirical DataSIAM Review, 2009
- On the equivalence between Non-negative Matrix Factorization and Probabilistic Latent Semantic IndexingComputational Statistics & Data Analysis, 2008
- Projected Gradient Methods for Nonnegative Matrix FactorizationNeural Computation, 2007
- Stochastic Learning2004
- Convergence Rate of Incremental Subgradient Algorithms2001
- Likelihood Ratio Tests for Model Selection and Non-Nested HypothesesEconometrica, 1989