Tensor decompositions for learning latent variable models
Preprint
Abstract
This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models---including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation---which exploits a certain tensor structure in their low-order observable moments (typically, of second- and third-order). Specifically, parameter estimation is reduced to the problem of extracting a certain (orthogonal) decomposition of a symmetric tensor derived from the moments; this decomposition can be viewed as a natural generalization of the singular value decomposition for matrices. Although tensor decompositions are generally intractable to compute, the decomposition of these specially structured tensors can be efficiently obtained by a variety of approaches, including power iterations and maximization approaches (similar to the case of matrices). A detailed analysis of a robust tensor power method is provided, establishing an analogue of Wedin's perturbation theorem for the singular vectors of matrices. This implies a robust and computationally tractable estimation approach for several popular latent variable models.Keywords
All Related Versions
This publication has 37 references indexed in Scilit:
- Binary Cumulant VarietiesAnnals of Combinatorics, 2012
- A spectral algorithm for learning Hidden Markov ModelsJournal of Computer and System Sciences, 2012
- Learning a Parallelepiped: Cryptanalysis of GGH and NTRU SignaturesJournal of Cryptology, 2008
- Algebraic factor analysis: tetrads, pentads and beyondProbability Theory and Related Fields, 2006
- Adaptive blind separation of independent sources: A deflation approachSignal Processing, 1995
- Independent component analysis, A new concept?Signal Processing, 1994
- Perturbation bounds in connection with singular value decompositionBIT Numerical Mathematics, 1972
- “Parallel Proportional Profiles” and other Principles for Determining the Choice of Factors by RotationPsychometrika, 1944
- Multiple Invariants and Generalized Rank of a P‐Way Matrix or TensorJournal of Mathematics and Physics, 1928
- The Expression of a Tensor or a Polyadic as a Sum of ProductsJournal of Mathematics and Physics, 1927