The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization
Authors:
Top Cited Papers
- 1 October 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8320-8329
Abstract
We introduce four new real-world distribution shift datasets consisting of changes in image style, image blurriness, geographic location, camera operation, and more. With our new datasets, we take stock of previously proposed methods for improving out-of-distribution robustness and put them to the test. We find that using larger models and artificial data augmentations can improve robustness on real-world distribution shifts, contrary to claims in prior work. We find improvements in artificial robustness benchmarks can transfer to real-world distribution shifts, contrary to claims in prior work. Motivated by our observation that data augmentations can help with real-world distribution shifts, we also introduce a new data augmentation method which advances the state-of-the-art and outperforms models pre-trained with 1000× more labeled data. Overall we find that some methods consistently help with distribution shifts in texture and local image statistics, but these methods do not help with some other distribution shifts like geographic changes. Our results show that future research must study multiple distribution shifts simultaneously, as we demonstrate that no evaluated method consistently improves robustness.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features2019
- DeepFashion2: A Versatile Benchmark for Detection, Pose Estimation, Segmentation and Re-Identification of Clothing Images2019
- Squeeze-and-Excitation Networks2018
- A Study and Comparison of Human and Deep Learning Recognition Performance under Visual Distortions2017
- Enhanced Deep Residual Networks for Single Image Super-Resolution2017
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Google Street View: Capturing the World at Street LevelComputer, 2010
- Recognition of pictorial representations by chimpanzees (Pan troglodytes)Animal Cognition, 2006
- Recognition of Line-Drawing Representations by a Chimpanzee (Pan troglodytes)The Journal of General Psychology, 1994
- Surface versus edge-based determinants of visual recognitionCognitive Psychology, 1988