Visual Attention Consistency Under Image Transforms for Multi-Label Image Classification
Authors:
- 1 June 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 729-739
Abstract
Human visual perception shows good consistency for many multi-label image classification tasks under certain spatial transforms, such as scaling, rotation, flipping and translation. This has motivated the data augmentation strategy widely used in CNN classifier training -- transformed images are included for training by assuming the same class labels as their original images. In this paper, we further propose the assumption of perceptual consistency of visual attention regions for classification under such transforms, i.e., the attention region for a classification follows the same transform if the input image is spatially transformed. While the attention regions of CNN classifiers can be derived as an attention heatmap in middle layers of the network, we find that their consistency under many transforms are not preserved. To address this problem, we propose a two-branch network with an original image and its transformed image as inputs and introduce a new attention consistency loss that measures the attention heatmap consistency between two branches. This new loss is then combined with multi-label image classification loss for network training. Experiments on three datasets verify the superiority of the proposed network by achieving new state-of-the-art classification performance.Keywords
This publication has 42 references indexed in Scilit:
- Understanding image representations by measuring their equivariance and equivalence2015
- Fully convolutional networks for semantic segmentation2015
- Matrix Completion for Weakly-Supervised Multi-Label Image ClassificationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- PANDA: Pose Aligned Networks for Deep Attribute Modeling2014
- NUS-WIDE2009
- Multi-Label ClassificationInternational Journal of Data Warehousing and Mining, 2007
- Distracted and confused?: Selective attention under loadTrends in Cognitive Sciences, 2005
- Learning multi-label scene classificationPattern Recognition, 2004
- Neural Mechanisms of Selective Visual AttentionAnnual Review of Neuroscience, 1995
- Selective Attention Gates Visual Processing in the Extrastriate CortexScience, 1985