Context‐aware relation enhancement and similarity reasoning for image‐text retrieval
Authors:
Open Access
- 30 January 2024
- journal article
- research article
- Published by Institution of Engineering and Technology (IET) in IET Computer Vision
- Vol. 18 (5) , 652-665
Abstract
Image‐text retrieval is a fundamental yet challenging task, which aims to bridge a semantic gap between heterogeneous data to achieve precise measurements of semantic similarity. The technique of fine‐grained alignment between cross‐modal features plays a key role in various successful methods that have been proposed. Nevertheless, existing methods cannot effectively utilise intra‐modal information to enhance feature representation and lack powerful similarity reasoning to get a precise similarity score. Intending to tackle these issues, a context‐aware Relation Enhancement and Similarity Reasoning model, called RESR, is proposed, which conducts both intra‐modal relation enhancement and inter‐modal similarity reasoning while considering the global‐context information. For intra‐modal relation enhancement, a novel context‐aware graph convolutional network is introduced to enhance local feature representations by utilising relation and global‐context information. For inter‐modal similarity reasoning, local and global similarity features are exploited by the bidirectional alignment of image and text, and the similarity reasoning is implemented among multi‐granularity similarity features. Finally, refined local and global similarity features are adaptively fused to get a precise similarity score. The experimental results show that our effective model outperforms some state‐of‐the‐art approaches, achieving average improvements of 2.5% and 6.3% in R@sum on the Flickr30K and MS‐COCO dataset.Keywords
This publication has 33 references indexed in Scilit:
- Dual-path Convolutional Image-Text Embeddings with Instance LossACM Transactions on Multimedia Computing, Communications, and Applications, 2020
- Cross-Modal Attention With Semantic Consistence for Image–Text MatchingIEEE Transactions on Neural Networks and Learning Systems, 2020
- Focus Your Attention2019
- Bi-Directional Spatial-Semantic Attention Networks for Image-Text MatchingIEEE Transactions on Image Processing, 2018
- Adversarial Cross-Modal Retrieval2017
- Hetero-Manifold Regularisation for Cross-Modal HashingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Grounded Compositional Semantics for Finding and Describing Images with SentencesTransactions of the Association for Computational Linguistics, 2014
- A new approach to cross-modal multimedia retrieval2010
- Bidirectional recurrent neural networksIEEE Transactions on Signal Processing, 1997
- Long Short-Term MemoryNeural Computation, 1997