A Tunable Loss Function for Robust Classification: Calibration, Landscape, and Generalization
- 21 April 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Information Theory
- Vol. 68 (9) , 6021-6051
Abstract
We introduce a tunable loss function called $\alpha $ -loss, parameterized by $\alpha \in (0,\infty]$ , which interpolates between the exponential loss ( $\alpha = 1/2$ ), the log-loss ( $\alpha = 1$ ), and the 0–1 loss ( $\alpha = \infty $ ), for the machine learning setting of classification. Theoretically, we illustrate a fundamental connection between $\alpha $ -loss and Arimoto conditional entropy, verify the classification-calibration of $\alpha $ -loss in order to demonstrate asymptotic optimality via Rademacher complexity generalization techniques, and build-upon a notion called strictly local quasi-convexity in order to quantitatively characterize the optimization landscape of $\alpha $ -loss. Practically, we perform class imbalance, robustness, and classification experiments on benchmark image datasets using convolutional-neural-networks. Our main practical conclusion is that certain tasks may benefit from tuning $\alpha $ -loss away from log-loss ( $\alpha = 1$ ), and to this end we provide simple heuristics for the practitioner. In particular, navigating the $\alpha $ hyperparameter can readily provide superior model robustness to label flips ( $\alpha > 1$ ) and sensitivity to imbalanced classes ( $\alpha < 1$ ).
Keywords
Funding Information
- NSF (CIF-1901243, CIF-1815361, CIF-2007688, DMS-2134256, SaTC-2031799, CNS-2003111)
- Programa de Apoyo a Proyectos de Investigación e Innovación Tecnológica (IA101021)
- Office of Naval Research (N00014-21-1-2615)
This publication has 41 references indexed in Scilit:
- Random classification noise defeats all convex potential boostersMachine Learning, 2009
- On surrogate loss functions and f-divergencesThe Annals of Statistics, 2009
- Fast learning rates for plug-in classifiersThe Annals of Statistics, 2007
- Convexity, Classification, and Risk BoundsJournal of the American Statistical Association, 2006
- A note on margin-based loss functions in classificationStatistics & Probability Letters, 2004
- Are Loss Functions All the Same?Neural Computation, 2004
- On the difficulty of approximately maximizing agreementsJournal of Computer and System Sciences, 2003
- A mathematical theory of communicationACM SIGMOBILE Mobile Computing and Communications Review, 2001
- Fair end-to-end window-based congestion controlIEEE/ACM Transactions on Networking, 2000
- Information-theoretical considerations on estimation problemsInformation and Control, 1971