Generalized Maximum Entropy for Supervised Classification
- 14 January 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Information Theory
- Vol. 68 (4) , 2530-2550
Abstract
The maximum entropy principle advocates to evaluate events’ probabilities using a distribution that maximizes entropy among those that satisfy certain expectations’ constraints. Such principle can be generalized for arbitrary decision problems where it corresponds to minimax approaches. This paper establishes a framework for supervised classification based on the generalized maximum entropy principle that leads to minimax risk classifiers (MRCs). We develop learning techniques that determine MRCs for general entropy functions and provide performance guarantees by means of convex optimization. In addition, we describe the relationship of the presented techniques with existing classification methods, and quantify MRCs performance in comparison with the proposed bounds and conventional methods.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (61871256)
- Spanish Ministry of Economy and Competitiveness MINECO (PID2019-105058GA-I00)
- Severo Ochoa Excellence Accreditation (SEV-2017-0718)
- Basque Government BERC (BERC 2018-2021)
- Ramón y Cajal grant (RYC-2016-19383)
- Basque Government ELKARTEK
This publication has 35 references indexed in Scilit:
- A Maximum Entropy Model Applied to Spatial and Temporal Correlations from Cortical NetworksIn VitroThe Journal of Neuroscience, 2008
- Probabilistic Forecasts, Calibration and SharpnessJournal of the Royal Statistical Society Series B: Statistical Methodology, 2007
- Convexity, Classification, and Risk BoundsJournal of the American Statistical Association, 2006
- Pattern Recognition and Machine Learning2006
- Statistical behavior and consistency of classification methods based on convex risk minimizationThe Annals of Statistics, 2004
- Maximum Entropy FundamentalsEntropy, 2001
- Minimum relative entropy and probabilistic inversion in groundwater hydrologyStochastic Environmental Research and Risk Assessment, 1998
- Efficient noise-tolerant learning from statistical queriesJournal of the ACM, 1998
- Why Least Squares and Maximum Entropy? An Axiomatic Approach to Inference for Linear Inverse ProblemsThe Annals of Statistics, 1991
- Information Theory and Statistical MechanicsPhysical Review B, 1957