Modality-Specific Cross-Modal Similarity Measurement With Recurrent Attention Network
Authors:
- 2 July 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 27 (11) , 5585-5599
Abstract
Nowadays, cross-modal retrieval plays an important role to flexibly find useful information across different modalities of data. Effectively measuring the similarity between different modalities of data is the key of cross-modal retrieval. Different modalities such as image and text have imbalanced and complementary relationship, and they contain unequal amount of information when describing the same semantics. For example, images often contain more details that cannot be demonstrated by textual descriptions and vice versa. Existing works based on Deep Neural Network (DNN) mostly construct one common space for different modalities, to find the latent alignments between them, which lose their exclusive modality-specific characteristics. Therefore, we propose modality-specific cross-modal similarity measurement (MCSM) approach by constructing the independent semantic space for each modality, which adopts an endto- end framework to directly generate modality-specific crossmodal similarity without explicit common representation. For each semantic space, modality-specific characteristics within one modality are fully exploited by recurrent attention network, while the data of another modality is projected into this space with attention based joint embedding, which utilizes the learned attention weights for guiding the fine-grained cross-modal correlation learning, and captures the imbalanced and complementary relationship between different modalities. Finally, the complementarity between the semantic spaces for different modalities is explored by adaptive fusion of the modality-specific cross-modal similarities to perform cross-modal retrieval. Experiments on the widely-used Wikipedia, Pascal Sentence, MS-COCO datasets as well as our constructed large-scale XMediaNet dataset verify the effectiveness of our proposed approach, outperforming 9 stateof- the-art methods.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (61771025, 61532005)
This publication has 32 references indexed in Scilit:
- Neighborhood Discriminant Hashing for Large-Scale Image RetrievalIEEE Transactions on Image Processing, 2015
- Cross-Modal Learning to Rank via Latent Joint RepresentationIEEE Transactions on Image Processing, 2015
- Semi-Supervised Cross-Media Feature Learning With Unified Patch Graph RegularizationIEEE Transactions on Circuits and Systems for Video Technology, 2015
- Learning Consistent Feature Representation for Cross-Modal Multimedia RetrievalIEEE Transactions on Multimedia, 2015
- Convolutional Neural Networks for Sentence Classification2014
- A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their SemanticsInternational Journal of Computer Vision, 2013
- On the Role of Correlation and Abstraction in Cross-Modal Multimedia RetrievalIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Canonical Correlation Analysis: An Overview with Application to Learning MethodsNeural Computation, 2004
- Long Short-Term MemoryNeural Computation, 1997
- RELATIONS BETWEEN TWO SETS OF VARIATESBiometrika, 1936