Soft Calibration Objectives for Neural Networks
Preprint
Abstract
Optimal decision making requires that classifiers produce uncertainty estimates consistent with their empirical accuracy. However, deep neural networks are often under- or over-confident in their predictions. Consequently, methods have been developed to improve the calibration of their predictive uncertainty both during training and post-hoc. In this work, we propose differentiable losses to improve calibration based on a soft (continuous) version of the binning operation underlying popular calibration-error estimators. When incorporated into training, these soft calibration losses achieve state-of-the-art single-model ECE across multiple datasets with less than 1% decrease in accuracy. For instance, we observe an 82% reduction in ECE (70% relative to the post-hoc rescaled ECE) in exchange for a 0.7% relative decrease in accuracy relative to the cross entropy baseline on CIFAR-100. When incorporated post-training, the soft-binning-based calibration error objective improves upon temperature scaling, a popular recalibration method. Overall, experiments across losses and datasets demonstrate that using calibration-sensitive procedures yield better uncertainty estimates under dataset shift than the standard practice of using a cross entropy loss and post-hoc recalibration methods.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Focal Loss for Dense Object DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Estimating Uncertainty Online Against an AdversaryProceedings of the AAAI Conference on Artificial Intelligence, 2017
- Deep Residual Learning for Image Recognition2016
- Wide Residual Networks2016
- Calibrating predictive model estimates to support personalized medicineJournal of the American Medical Informatics Association, 2012
- Using confidence intervals for graphically based data interpretation.Canadian Journal of Experimental Psychology / Revue canadienne de psychologie expérimentale, 2003