Bi-Directional Spatial-Semantic Attention Networks for Image-Text Matching
- 19 November 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 28 (4) , 2008-2020
Abstract
Image-text matching by deep models has recently made remarkable achievements in many tasks, such as image caption and image search. A major challenge of matching the image and text lies in that they usually have complicated underlying relations between them and simply modeling the relations may lead to suboptimal performance. In this paper, we develop a novel approach Bi-directional Spatial-Semantic Attention Networks (BSSAN), which leverages both the word to regions (W2R) relation and image object to words (O2W) relation in a holistic deep framework for more effectively matching. Specifically, to effectively encode the W2R relation, we adopt LSTM with bilinear attention function to infer the image regions which are more related to the particular words, which is referred as the W2R attention network. On the other side, the O2W attention network is proposed to discover the semanticallyclose words for each visual object in the image, i.e., the visual object to words (O2W) relation. Then a deep model unifying both of the two directional attention networks into a holistic learning framework is proposed to learn the matching scores of image and text pairs. Compared to existing image-text matching methods, our approach achieves state-of-the-art performance on the datasets of Flickr30K and MSCOCO.Keywords
Funding Information
- Natural Science Foundation of Beijing Municipality (4182037)
- National Natural Science Foundation of China (U1636210, U1636211)
- State Key Laboratory of Software Development Environment (SKLSDE-2017ZX-19)
This publication has 29 references indexed in Scilit:
- Multimodal Convolutional Neural Networks for Matching Image and Sentence2015
- Deep visual-semantic alignments for generating image descriptions2015
- Deep correlation for matching images and text2015
- BSIFT: Toward Data-Independent Codebook for Large Scale Image SearchIEEE Transactions on Image Processing, 2015
- Effective Approaches to Attention-based Neural Machine Translation2015
- Contextual Hashing for Large-Scale Image SearchIEEE Transactions on Image Processing, 2014
- Glove: Global Vectors for Word Representation2014
- A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their SemanticsInternational Journal of Computer Vision, 2013
- The Pascal Visual Object Classes (VOC) ChallengeInternational Journal of Computer Vision, 2009
- Canonical Correlation Analysis: An Overview with Application to Learning MethodsNeural Computation, 2004