SST: Spatial and Semantic Transformers for Multi-Label Image Recognition
- 11 March 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 31 (10577149) , 2570-2583
Abstract
Multi-label image recognition has attracted considerable research attention and achieved great success in recent years. Capturing label correlations is an effective manner to advance the performance of multi-label image recognition. Two types of label correlations were principally studied, i.e., the spatial and semantic correlations. However, in the literature, previous methods considered only either of them. In this work, inspired by the great success of Transformer, we propose a plug-and-play module, named the Spatial and Semantic Transformers (SST), to simultaneously capture spatial and semantic correlations in multi-label images. Our proposal is mainly comprised of two independent transformers, aiming to capture the spatial and semantic correlations respectively. Specifically, our Spatial Transformer is designed to model the correlations between features from different spatial positions, while the Semantic Transformer is leveraged to capture the co-existence of labels without manually defined rules. Other than methodological contributions, we also prove that spatial and semantic correlations complement each other and deserve to be leveraged simultaneously in multi-label image recognition. Benefitting from the Transformer’s ability to capture long-range correlations, our method remarkably outperforms state-of-the-art methods on four popular multi-label benchmark datasets. In addition, extensive ablation studies and visualizations are provided to validate the essential components of our method.Keywords
Funding Information
- Zhejiang Provincial Natural Science Foundation of China (LQ22F020006)
- National Natural Science Foundation of China (61922064, U2033210, 62101387)
This publication has 31 references indexed in Scilit:
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Beyond Object Proposals: Random Crop Pooling for Multi-Label Image RecognitionIEEE Transactions on Image Processing, 2016
- Deep Residual Learning for Image Recognition2016
- CNN-RNN: A Unified Framework for Multi-label Image Classification2016
- Learning Structured Inference Neural Networks with Label Relations2016
- Exploit Bounding Box Annotations for Multi-Label Object Recognition2016
- Learning Deep Features for Discriminative Localization2016
- HCP: A Flexible CNN Framework for Multi-Label Image ClassificationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- NUS-WIDE2009
- Long Short-Term MemoryNeural Computation, 1997