The MEE Principle in Data Classification: A Perceptron-Based Analysis
- 1 October 2010
- journal article
- Published by MIT Press in Neural Computation
- Vol. 22 (10) , 2698-2728
Abstract
This letter focuses on the issue of whether risk functionals derived from information-theoretic principles, such as Shannon or Rényi's entropies, are able to cope with the data classification problem in both the sense of attaining the risk functional minimum and implying the minimum probability of error allowed by the family of functions implemented by the classifier, here denoted by min Pe. The analysis of this so-called minimization of error entropy (MEE) principle is carried out in a single perceptron with continuous activation functions, yielding continuous error distributions. In spite of the fact that the analysis is restricted to single perceptrons, it reveals a large spectrum of behaviors that MEE can be expected to exhibit in both theory and practice. In what concerns the theoretical MEE, our study clarifies the role of the parameters controlling the perceptron activation function (of the squashing type) in often reaching the minimum probability of error. Our study also clarifies the role of the kernel density estimator of the error density in achieving the minimum probability of error in practice.Keywords
This publication has 18 references indexed in Scilit:
- Correntropy: Properties and Applications in Non-Gaussian Signal ProcessingIEEE Transactions on Signal Processing, 2007
- Error Entropy Minimization for LSTM TrainingLecture Notes in Computer Science, 2006
- Convergence properties and data efficiency of the minimum error entropy criterion in adaline trainingIEEE Transactions on Signal Processing, 2003
- Blind source separation using Renyi's α-marginal entropiesNeurocomputing, 2002
- Information theoretic clusteringIEEE Transactions on Pattern Analysis and Machine Intelligence, 2002
- An error-entropy minimization algorithm for supervised training of nonlinear adaptive systemsIEEE Transactions on Signal Processing, 2002
- Blind source separation using Renyi's mutual informationIEEE Signal Processing Letters, 2001
- Convergence Properties of the Nelder--Mead Simplex Method in Low DimensionsSIAM Journal on Optimization, 1998
- Adaptive Mixtures of Local ExpertsNeural Computation, 1991
- Self-organization in a perceptual networkComputer, 1988