CausMatch: Causal Matching Learning With Counterfactual Preference Framework for Cross-Modal Retrieval
Open Access
- 15 January 2025
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 12734-12745
Abstract
Cross-modal retrieval exhibits significant promise within the realm of multimedia analysis. Numerous sophisticated techniques have gained widespread adoption for harnessing attention mechanisms to facilitate cross-modal correspondence in matching tasks. However, most existing methods learn cross-modal attention based on conventional likelihood, leading to the fine-gained matching from regions and words containing numerous invalid local relationships and false global connections. This phenomenon will bring negative effects on alignment. Different from these methods, we propose a novel Causal Matching Learning (CausMatch) with counterfactual preference framework for cross-modal retrieval in this paper. This work seeks to ascertain the matching relation by incorporating a counterfactual causality preference, enhancing the quality of attention and providing a robust supervisory signal for the learning process are pivotal objectives. This study specifically investigates the influence of acquired visual and textual attention on network predictions by employing counterfactual intervention as a method of scrutiny. This approach aims to discern and analyze the consequential effects on the learning process. Our approach involves maximizing the positive influence to incentivize the network to assimilate more pertinent relationships and connections conducive to fine-grained cross-modal retrieval. The proposed CausMatch model’s effectiveness is systematically substantiated through a comprehensive series of experiments conducted on two widely recognized benchmark datasets, namely MS-COCO and Flickr30K. The results unequivocally demonstrate its superiority over existing state-of-the-art methods, underscoring its robust performance in cross-modal retrieval tasks.Keywords
Funding Information
- Development and Application Project of Ship Computer Aided Engineering Software
- National Natural Science Foundation of China (62201419)
This publication has 39 references indexed in Scilit:
- Cross-Modal Retrieval: A Review of Methodologies, Datasets, and Future PerspectivesIEEE Access, 2024
- CREAMY: Cross-Modal Recipe Retrieval By Avoiding Matching ImperfectlyIEEE Access, 2024
- Supervised Consensus Anchor Graph Hashing for Cross Modal RetrievalIEEE Access, 2024
- Image-Text Connection: Exploring the Expansion of the Diversity Within Joint Feature Space Similarity ScoresIEEE Access, 2023
- Supervised Matrix Factorization Hashing With Quantitative Loss for Image-Text SearchIEEE Access, 2020
- Adversarial Attentive Multi-Modal Embedding Learning for Image-Text MatchingIEEE Access, 2020
- Multi-Modal Memory Enhancement Attention Network for Image-Text MatchingIEEE Access, 2020
- Stacked Cross Attention for Image-Text Matching2018
- Deep visual-semantic alignments for generating image descriptions2015
- Microsoft COCO: Common Objects in Context2014