ReAL : Improving Image-Text Retrieval with Authentic Negative Repository Learning
Open Access
- 12 April 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
Abstract
Current methods for image-text retrieval commonly propose various fusion modules to achieve robust visual-textual alignment, primarily relying on in-batch learning to guide the matching process. Some follow-up methods seek to enlarge the number of negative samples to boost image-text contrastive learning. However, these methods often face challenges posed by semantic-consistent negatives, i.e., negatives samples that share correspondence with the ground truth, leading to confusion in learning cross-modal semantics. To address this issue, we propose a novel Re trieve with A uthentic negative repository L earning ( ReAL ) method, which constructs a specific Authentic Negative Repository filled with valuable negative sample pairs. By introducing a Unique Negative Filter with a Discriminative Triplet Ranking Loss, ReAL effectively filters out the semantic-consistent negatives through similarity distribution analysis and threshold learning. Moreover, existing fusion paradigms suffer from intricate use of fine-grained representations from word- and region-level instances to progressively refine the fused embedding. In this paper, we propose a lightweight Cluster Refinement Module to exploit cross-modal semantics in a 1-way-1-out paradigm. Each visual-textual alignment can spontaneously uncover correlations with adjacent alignments through aggregation and re-allocation, without the need for a redundant and cost-inefficient refinement stage. Furthermore, ReAL employs dual momentum encoders with two memory banks, expanding the selection range of the Authentic Negative Repository to include a broader set of negatives. Extensive experiments conducted on Flickr30K, MS-COCO, and the augmented Flickr30K (with more hard negatives) demonstrate the superiority and robustness of ReAL, while also showcasing its significantly reduced inference time compared to other competitive baselines.Keywords
This publication has 48 references indexed in Scilit:
- CODER: Coupled Diversity-Sensitive Momentum Contrastive Learning for Image-Text Retrieval2022
- Unified Visual-Semantic Embeddings: Bridging Vision and Language With Structured Meaning Representations2019
- Deep Cross-Modal Projection Learning for Image-Text Matching2018
- Stacked Cross Attention for Image-Text Matching2018
- Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models2018
- Dual Attention Networks for Multimodal Reasoning and Matching2017
- Cross-Modal Supervision for Learning Active Speaker Detection in Video2016
- Deep Residual Learning for Image Recognition2016
- FaceNet: A unified embedding for face recognition and clustering2015
- Microsoft COCO: Common Objects in Context2014