On the Smoothed Minimum Error Entropy Criterion
Open Access
- 12 November 2012
- Vol. 14 (11) , 2311-2323
Abstract
Recent studies suggest that the minimum error entropy (MEE) criterion can outperform the traditional mean square error criterion in supervised machine learning, especially in nonlinear and non-Gaussian situations. In practice, however, one has to estimate the error entropy from the samples since in general the analytical evaluation of error entropy is not possible. By the Parzen windowing approach, the estimated error entropy converges asymptotically to the entropy of the error plus an independent random variable whose probability density function (PDF) corresponds to the kernel function in the Parzen method. This quantity of entropy is called the smoothed error entropy, and the corresponding optimality criterion is named the smoothed MEE (SMEE) criterion. In this paper, we study theoretically the SMEE criterion in supervised machine learning where the learning machine is assumed to be nonparametric and universal. Some basic properties are presented. In particular, we show that when the smoothing factor is very small, the smoothed error entropy equals approximately the true error entropy plus a scaled version of the Fisher information of error. We also investigate how the smoothing factor affects the optimal solution. In some special situations, the optimal solution under the SMEE criterion does not change with increasing smoothing factor. In general cases, when the smoothing factor tends to infinity, minimizing the smoothed error entropy will be approximately equivalent to minimizing error variance, regardless of the conditional PDF and the kernel.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Minimum entropy filtering for multivariate stochastic systems with non-Gaussian noisesIEEE Transactions on Automatic Control, 2006
- Convergence properties and data efficiency of the minimum error entropy criterion in adaline trainingIEEE Transactions on Signal Processing, 2003
- Generalized information potential criterion for adaptive system trainingIEEE Transactions on Neural Networks, 2002
- An error-entropy minimization algorithm for supervised training of nonlinear adaptive systemsIEEE Transactions on Signal Processing, 2002
- Entropy minimization for supervised digital communications channel equalizationIEEE Transactions on Signal Processing, 2002
- Simple proofs of two results on convolutions of unimodal distributionsStatistics & Probability Letters, 1998
- Optimal state estimation for stochastic systems: an information theoretic approachIEEE Transactions on Automatic Control, 1997
- Minimum entropy of error principle in estimationInformation Sciences, 1994
- Linear prediction, filtering, and smoothing: An information-theoretic approachInformation Sciences, 1979
- Learning and Information TheoryInternational Journal of Neuroscience, 1972