Bayesian Invariant Risk Minimization
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 16000-16009
Abstract
Generalization under distributional shift is an open challenge for machine learning. Invariant Risk Minimization (IRM) is a promising framework to tackle this issue by extracting invariant features. However, despite the potential and popularity of IRM, recent works have reported negative results of it on deep models. We argue that the failure can be primarily attributed to deep models' tendency to overfit the data. Specifically, our theoretical analysis shows that IRM degenerates to empirical risk minimization (ERM) when overfitting occurs. Our empirical evidence also provides supports: IRM methods that work well in typical settings significantly deteriorate even if we slightly enlarge the model size or lessen the training data. To alleviate this issue, we propose Bayesian Invariant Risk Min-imization (BIRM) by introducing Bayesian inference into the IRM. The key motivation is to estimate the penalty of IRM based on the posterior distribution of classifiers (as opposed to a single classifier), which is much less prone to overfitting. Extensive experimental results on four datasets demonstrate that BIRM consistently outperforms the existing IRM baselines significantly.Keywords
This publication has 16 references indexed in Scilit:
- Vocabulary-Informed Zero-Shot and Open-Set LearningIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019
- Variational Inference: A Review for StatisticiansJournal of the American Statistical Association, 2017
- Handbook of Uncertainty Quantification2017
- Causal Inference by using Invariant Prediction: Identification and Confidence IntervalsJournal of the Royal Statistical Society Series B: Statistical Methodology, 2016
- Deep Residual Learning for Image Recognition2016
- Probabilistic machine learning and artificial intelligenceNature, 2015
- Effective Approaches to Attention-based Neural Machine Translation2015
- Speech recognition with deep recurrent neural networks2013
- Suboptimality of Penalized Empirical Risk Minimization in Classification2007
- Keeping the neural networks simple by minimizing the description length of the weights1993