Fine-grained Image-text Matching by Cross-modal Hard Aligning Network
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 19275-19284
Abstract
Current state-of-the-art image-text matching methods implicitly align the visual-semantic fragments, like regions in images and words in sentences, and adopt cross-attention mechanism to discover fine-grained cross-modal semantic correspondence. However, the cross-attention mechanism may bring redundant or irrelevant region-word alignments, degenerating retrieval accuracy and limiting efficiency. Although many researchers have made progress in mining meaningful alignments and thus improving accuracy, the problem of poor efficiency remains unresolved. In this work, we propose to learn fine-grained image-text matching from the perspective of information coding. Specifically, we suggest a coding framework to explain the fragments aligning process, which provides a novel view to reexamine the cross-attention mechanism and analyze the problem of redundant alignments. Based on this framework, a Cross-modal Hard Aligning Network (CHAN) is designed, which comprehensively exploits the most relevant region-word pairs and eliminates all other alignments. Extensive experiments conducted on two public datasets, MS-COCO and Flickr30K, verify that the relevance of the most associated word-region pairs is discriminative enough as an indicator of the image-text similarity, with superior accuracy and efficiency over the state-of-the-art approaches on the bidirectional image and text retrieval tasks. Our code will be available at https://github.com/ppanzx/CHAN.Keywords
Funding Information
- National Natural Science Foundation of China (62172356)
This publication has 27 references indexed in Scilit:
- Adversarial Cross-Modal Retrieval2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Deep Residual Learning for Image Recognition2016
- Deep visual-semantic alignments for generating image descriptions2015
- Glove: Global Vectors for Word Representation2014
- Feature Coding in Image Classification: A Comprehensive StudyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Visual Word AmbiguityIEEE Transactions on Pattern Analysis and Machine Intelligence, 2009
- Kernel Codebooks for Scene Categorization2008
- Video Google: a text retrieval approach to object matching in videos2003