Not Using the Car to See the Sidewalk — Quantifying and Controlling the Effects of Context in Classification and Segmentation
Authors:
- 1 June 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8210-8218
Abstract
Importance of visual context in scene understanding tasks is well recognized in the computer vision community. However, to what extent the computer vision models are dependent on the context to make their predictions is unclear. A model overly relying on context will fail when encountering objects in different contexts than in training data and hence it is important to identify these dependencies before we can deploy the models in the real-world. We propose a method to quantify the sensitivity of black-box vision models to visual context by editing images to remove selected objects and measuring the response of the target models. We apply this methodology on two tasks, image classification and semantic segmentation, and discover undesirable dependency between objects and context, for example that ``sidewalk'' segmentation is very sensitive to the presence of ``cars'' in the image. We propose an object removal based data augmentation solution to mitigate this dependency and increase the robustness of classification and segmentation models to contextual variations. Our experiments show that the proposed data augmentation helps these models improve the performance in out-of-context scenarios, while preserving the performance on regular data.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- "Why Should I Trust You?"2016
- Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks2016
- Is object localization for free? - Weakly-supervised learning with convolutional neural networks2015
- What do 15,000 object categories tell us about classifying and localizing actions?2015
- Fully convolutional networks for semantic segmentation2015
- The Role of Context for Object Detection and Semantic Segmentation in the Wild2014
- Quantifying the role of context in visual object recognitionVisual Cognition, 2013
- Exploring Tiny Images: The Roles of Appearance and Contextual Information for Machine and Human Object RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2011
- Discriminative Models for Multi-Class Object LayoutInternational Journal of Computer Vision, 2011
- Using the forest to see the treesCommunications of the ACM, 2010