Fast Tuning of Topic Models: An Application of Rényi Entropy and Renormalization Theory
Open Access
- 17 November 2019
- conference paper
- Published by MDPI AG in Proceedings
- Vol. 46 (1) , 5
Abstract
In practice, the critical step in building machine learning models of big data (BD) is costly in terms of time and the computing resources procedure of parameter tuning with a grid search. Due to the size, BD are comparable to mesoscopic physical systems. Hence, methods of statistical physics could be applied to BD. The paper shows that topic modeling demonstrates self-similar behavior under the condition of a varying number of clusters. Such behavior allows using a renormalization technique. The combination of a renormalization procedure with the Rényi entropy approach allows for fast searching of the optimal number of clusters. In this paper, the renormalization procedure is developed for the Latent Dirichlet Allocation (LDA) model with a variational Expectation-Maximization algorithm. The experiments were conducted on two document collections with a known number of clusters in two languages. The paper presents results for three versions of the renormalization procedure: (1) a renormalization with the random merging of clusters, (2) a renormalization based on minimal values of Kullback–Leibler divergence and (3) a renormalization with merging clusters with minimal values of Rényi entropy. The paper shows that the renormalization procedure allows finding the optimal number of topics 26 times faster than grid search without significant loss of quality.Keywords
Funding Information
- Basic Research Program at the National Research University Higher School of Economics (HSE)
This publication has 9 references indexed in Scilit:
- Estimating Topic Modeling Performance with Sharma–Mittal EntropyEntropy, 2019
- Fractal approach for determining the optimal number of topics in the field of topic modeling.Journal of Physics: Conference Series, 2019
- Application of Rényi and Tsallis entropies to topic modeling optimizationPhysica A: Statistical Mechanics and its Applications, 2018
- A Model of Text for Experimentation in the Social SciencesJournal of the American Statistical Association, 2016
- A density-based method for adaptive LDA model selectionNeurocomputing, 2009
- Constrained Clustering2008
- The world according to Rényi: thermodynamics of multifractal systemsAnnals of Physics, 2004
- The world according to R nyi: thermodynamics of multifractal systemsAnnals of Physics, 2004
- Probabilistic latent semantic indexing1999