Asymptotic Convergence Properties of the EM Algorithm for Mixture of Experts
Authors:
- 1 August 2011
- journal article
- Published by MIT Press in Neural Computation
- Vol. 23 (8) , 2140-2168
Abstract
Mixture of experts (ME) is a modular neural network architecture for supervised classification. The double-loop expectation-maximization (EM) algorithm has been developed for learning the parameters of the ME architecture, and the iteratively reweighted least squares (IRLS) algorithm and the Newton-Raphson algorithm are two popular schemes for learning the parameters in the inner loop or gating network. In this letter, we investigate asymptotic convergence properties of the EM algorithm for ME using either the IRLS or Newton-Raphson approach. With the help of an overlap measure for the ME model, we obtain an upper bound of the asymptotic convergence rate of the EM algorithm in each case. Moreover, we find that for the Newton approach as a specific Newton-Raphson approach to learning the parameters in the inner loop, the upper bound of asymptotic convergence rate of the EM algorithm locally around the true solution Θ* is [ ], where ϵ>0 is an arbitrarily small number, o(x) means that it is a higher-order infinitesimal as x → 0, and e(Θ*) is a measure of the average overlap of the ME model. That is, as the average overlap of the true ME model with large sample tends to zero, the EM algorithm with the Newton approach to learning the parameters in the inner loop tends to be asymptotically superlinear. Finally, we substantiate our theoretical results by simulation experiments.Keywords
This publication has 12 references indexed in Scilit:
- On the correct convergence of the EM algorithm for Gaussian mixturesPattern Recognition, 2005
- Asymptotic convergence properties of the EM algorithm with respect to the overlap in the mixtureNeurocomputing, 2005
- Using the EM Algorithm to Train Neural Networks: Misconceptions and a New Algorithm for Multiclass ClassificationIEEE Transactions on Neural Networks, 2004
- Asymptotic Convergence Rate of the EM Algorithm for Gaussian MixturesNeural Computation, 2000
- Improved learning algorithms for mixture of experts in multiclass classificationNeural Networks, 1999
- Comparative Analysis on Convergence Rates of The EM Algorithm and Its Two Modifications for Gaussian MixturesNeural Processing Letters, 1997
- On Convergence Properties of the EM Algorithm for Gaussian MixturesNeural Computation, 1996
- On the Rate of Convergence of the ECM AlgorithmThe Annals of Statistics, 1994
- Hierarchical Mixtures of Experts and the EM AlgorithmNeural Computation, 1994
- Adaptive Mixtures of Local ExpertsNeural Computation, 1991