Efficient Computation of Normalized Maximum Likelihood Codes for Gaussian Mixture Models With Its Applications to Clustering
- 2 August 2013
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Information Theory
- Vol. 59 (11) , 7718-7727
Abstract
This paper addresses the issue of estimating from a given data sequence the number of mixture components for a Gaussian mixture model(GMM). Our approach is to compute the normalized maximum likelihood (NML) code length for the data sequence relative to a GMM, then to find the mixture size that attains the minimum of the NML on the basis of the minimum description length principle. For finite domains, Kontkanen and Myllymäki proposed a method for efficient computation of the NML code length for specific models, however, for general classes over infinite domains, it has remained open how we compute the NML code length efficiently. We first propose a general method for calculating the NML code length for a general exponential family. Then, we apply it to the efficient computation of the NML code length for a GMM. The key idea is to restrict the data domain in combination with the technique of employing a generating function for computing the normalization term for a GMM. We use artificial datasets to empirically demonstrate that our estimate of the mixture size converges to the true one significantly faster than other criteria.Keywords
This publication has 14 references indexed in Scilit:
- Optimal Estimation of Parameters2012
- Variable selection in linear regression: Several approaches based on normalized maximum likelihoodSignal Processing, 2011
- Efficient computation of normalized maximum likelihood coding for Gaussian mixtures with its applications to optimal clustering2011
- A linear-time algorithm for computing the multinomial stochastic complexityInformation Processing Letters, 2007
- Information and Complexity in Statistical Modeling2007
- MDL denoisingIEEE Transactions on Information Theory, 2000
- Fisher information and stochastic complexityIEEE Transactions on Information Theory, 1996
- Estimating the Dimension of a ModelThe Annals of Statistics, 1978
- A new look at the statistical model identificationIEEE Transactions on Automatic Control, 1974
- An Information Measure for ClassificationThe Computer Journal, 1968