OccamNets: Mitigating Dataset Bias by Favoring Simpler Hypotheses
Preprint
Abstract
Dataset bias and spurious correlations can significantly impair generalization in deep neural networks. Many prior efforts have addressed this problem using either alternative loss functions or sampling strategies that focus on rare patterns. We propose a new direction: modifying the network architecture to impose inductive biases that make the network robust to dataset bias. Specifically, we propose OccamNets, which are biased to favor simpler solutions by design. OccamNets have two inductive biases. First, they are biased to use as little network depth as needed for an individual example. Second, they are biased toward using fewer image locations for prediction. While OccamNets are biased toward simpler hypotheses, they can learn more complex hypotheses if necessary. In experiments, OccamNets outperform or rival state-of-the-art methods run on architectures that do not incorporate these inductive biases. Furthermore, we demonstrate that when the state-of-the-art debiasing methods are combined with OccamNets results further improve.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Attention mechanisms in computer vision: A surveyComputational Visual Media, 2022
- Why Should We Add Early Exits to Neural Networks?Cognitive Computation, 2020
- REPAIR: Removing Representation Bias by Dataset Resampling2019
- Learning Not to Learn: Training Deep Neural Networks With Biased Data2019
- Class-Balanced Loss Based on Effective Number of Samples2019
- Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering2018
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Going deeper with convolutions2015
- Microsoft COCO: Common Objects in Context2014
- Unbiased look at dataset bias2011