Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
Preprint
Abstract
Bayesian neural networks (BNN) and deep ensembles are principled approaches to estimate the predictive uncertainty of a deep learning model. However their practicality in real-time, industrial-scale applications are limited due to their heavy memory and inference cost. This motivates us to study principled approaches to high-quality uncertainty estimation that require only a single deep neural network (DNN). By formalizing the uncertainty quantification as a minimax learning problem, we first identify input distance awareness, i.e., the model's ability to quantify the distance of a testing example from the training data in the input space, as a necessary condition for a DNN to achieve high-quality (i.e., minimax optimal) uncertainty estimation. We then propose Spectral-normalized Neural Gaussian Process (SNGP), a simple method that improves the distance-awareness ability of modern DNNs, by adding a weight normalization step during training and replacing the output layer with a Gaussian process. On a suite of vision and language understanding tasks and on modern architectures (Wide-ResNet and BERT), SNGP is competitive with deep ensembles in prediction, calibration and out-of-domain detection, and outperforms the other single-model approaches.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Regularisation of neural networks by enforcing Lipschitz continuityMachine Learning, 2020
- Out-of-Domain Detection for Natural Language Understanding in Dialog SystemsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020
- User Utterance Acquisition for Training Task-Oriented Bots: A Review of Challenges, Techniques and OpportunitiesIEEE Internet Computing, 2020
- Why ReLU Networks Yield High-Confidence Predictions Far Away From the Training Data and How to Mitigate the Problem2019
- Residual Networks as Flows of DiffeomorphismsJournal of Mathematical Imaging and Vision, 2019
- Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection2018
- Robust Large Margin Deep Neural NetworksIEEE Transactions on Signal Processing, 2017
- Probabilism, entropies and strictly proper scoring rulesInternational Journal of Approximate Reasoning, 2015
- Reliability, sufficiency, and the decomposition of proper scoresQuarterly Journal of the Royal Meteorological Society, 2009
- Probabilistic Forecasts, Calibration and SharpnessJournal of the Royal Statistical Society Series B: Statistical Methodology, 2007