Abstract
We introduce a tunable loss function called $\alpha $ -loss, parameterized by $\alpha \in (0,\infty]$ , which interpolates between the exponential loss ( $\alpha = 1/2$ ), the log-loss ( $\alpha = 1$ ), and the 0–1 loss ( $\alpha = \infty $ ), for the machine learning setting of classification. Theoretically, we illustrate a fundamental connection between $\alpha $ -loss and Arimoto conditional entropy, verify the classification-calibration of $\alpha $ -loss in order to demonstrate asymptotic optimality via Rademacher complexity generalization techniques, and build-upon a notion called strictly local quasi-convexity in order to quantitatively characterize the optimization landscape of $\alpha $ -loss. Practically, we perform class imbalance, robustness, and classification experiments on benchmark image datasets using convolutional-neural-networks. Our main practical conclusion is that certain tasks may benefit from tuning $\alpha $ -loss away from log-loss ( $\alpha = 1$ ), and to this end we provide simple heuristics for the practitioner. In particular, navigating the $\alpha $ hyperparameter can readily provide superior model robustness to label flips ( $\alpha > 1$ ) and sensitivity to imbalanced classes ( $\alpha < 1$ ).
Funding Information
  • NSF (CIF-1901243, CIF-1815361, CIF-2007688, DMS-2134256, SaTC-2031799, CNS-2003111)
  • Programa de Apoyo a Proyectos de Investigación e Innovación Tecnológica (IA101021)
  • Office of Naval Research (N00014-21-1-2615)

This publication has 41 references indexed in Scilit: