Learning Semantic Relationship among Instances for Image-Text Matching
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 15159-15168
Abstract
Image-text matching, a bridge connecting image and language, is an important task, which generally learns a holistic cross-modal embedding to achieve a high-quality semantic alignment between the two modalities. However, previous studies only focus on capturing fragment-level relation within a sample from a particular modality, e.g., salient regions in an image or text words in a sentence, where they usually pay less attention to capturing instance-level interactions among samples and modalities, e.g., multiple images and texts. In this paper, we argue that sample relations could help learn subtle differences for hard negative instances, and thus transfer shared knowledge for infrequent samples should be promising in obtaining better holistic embeddings. Therefore, we propose a novel hierarchical relation modeling framework (HREM), which explicitly capture both fragment-and instance-level relations to learn discriminative and robust cross-modal embeddings. Extensive experiments on Flickr30K and MS-COCO show our proposed method out-performs the state-of-the-art ones by 4%-10% in terms of rSum. Our code is available at https://github.com/CrossmodalGroup/HREM.Keywords
Funding Information
- National Natural Science Foundation of China (62222212,62121002)
This publication has 44 references indexed in Scilit:
- Graph Structured Network for Image-Text Matching2020
- Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval2020
- Learning Fragment Self-Attention Embeddings for Image-Text Matching2019
- CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features2019
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Deep Residual Learning for Image Recognition2016
- From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptionsTransactions of the Association for Computational Linguistics, 2014
- Glove: Global Vectors for Word Representation2014
- Bidirectional recurrent neural networksIEEE Transactions on Signal Processing, 1997