Efficient Token-Guided Image-Text Retrieval With Consistent Multimodal Contrastive Training
- 20 June 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 32 (10577149) , 3622-3633
Abstract
Image-text retrieval is a central problem for understanding the semantic relationship between vision and language, and serves as the basis for various visual and language tasks. Most previous works either simply learn coarse-grained representations of the overall image and text, or elaborately establish the correspondence between image regions or pixels and text words. However, the close relations between coarse- and fine-grained representations for each modality are important for image-text retrieval but almost neglected. As a result, such previous works inevitably suffer from low retrieval accuracy or heavy computational cost. In this work, we address image-text retrieval from a novel perspective by combining coarse- and fine-grained representation learning into a unified framework. This framework is consistent with human cognition, as humans simultaneously pay attention to the entire sample and regional elements to understand the semantic content. To this end, a Token-Guided Dual Transformer (TGDT) architecture which consists of two homogeneous branches for image and text modalities, respectively, is proposed for image-text retrieval. The TGDT incorporates both coarse- and fine-grained retrievals into a unified framework and beneficially leverages the advantages of both retrieval approaches. A novel training objective called Consistent Multimodal Contrastive (CMC) loss is proposed accordingly to ensure the intra- and inter-modal semantic consistencies between images and texts in the common embedding space. Equipped with a two-stage inference method based on the mixed global and local cross-modal similarity, the proposed method achieves state-of-the-art retrieval performances with extremely low inference time when compared with representative recent approaches. Code is publicly available: github.com/LCFractal/TGDT.Keywords
All Related Versions
Funding Information
- Southeast University Start-Up Grant for New Faculty (RF1028623063)
- National Key Research and Development Program of China (2022ZD0117900)
- National Natural Science Foundation of China (62236010, 62276261)
This publication has 49 references indexed in Scilit:
- Stacked Cross Attention for Image-Text Matching2018
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2018
- Beyond Joints: Learning Representations From Primitive Geometries for Skeleton-Based Action Recognition and DetectionIEEE Transactions on Image Processing, 2018
- Learning Two-Branch Neural Networks for Image-Text Matching TasksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Sampling Matters in Deep Embedding Learning2017
- Large-Scale Image Retrieval with Attentive Deep Local Features2017
- Hierarchical Multimodal LSTM for Dense Visual-Semantic Embedding2017
- Deep Visual-Semantic Alignments for Generating Image DescriptionsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Learning Deep Structure-Preserving Image-Text Embeddings2016