Structures Aware Fine-Grained Contrastive Adversarial Hashing for Cross-Media Retrieval
- 29 January 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Knowledge and Data Engineering
- Vol. 36 (7) , 3514-3528
Abstract
Deep cross-media hashing provides an efficient semantic representation learning solution for large-scale cross-media retrieval. The existing methods only consider the inter-media or intra-media semantic association learning, ignore the guiding of semantic structure information, and have weak reasoning ability for implicit fine-grained semantic associations. To tackle this problem, we propose a novel structures aware fine-grained contrastive adversarial hashing method for cross-media retrieval. A novel cross-media contrastive adversarial hash network is constructed for the first time, which integrates the cross-media and intra-media contrastive learning and multi-modal adversarial learning, aiming at maximizing the semantic association between different modalities, and improving the semantic discrimination and consistency of cross-media unified hash representation, thereby the inter-media and intra-media semantic preserving ability can be well enhanced; A fine-grained cross-media semantic feature learning method based on fine-grained semantic reasoning with transformers is proposed, which captures fine-grained salient features of different modalities for semantic association learning, and enhances the reasoning ability of fine-grained implicit semantic association; A semantic label graph convolutional network guided cross-media semantic association learning strategy is proposed, which makes full use of semantic structure information to enhance the learning ability of implicit cross-media semantic associations. Extensive experiments on several large-scale cross-media benchmark datasets demonstrate that the proposed method outperforms the state-of-the-art methods.Keywords
Funding Information
- National Natural Science Foundation of China (62192784, U22B2038, 62172056, 62272058)
- 8th Young Elite Scientists Sponsorship Program by CAST (2022QNRC001)
This publication has 56 references indexed in Scilit:
- Local-Maximum-and-Minimum-Preserving Solution Remapping Technique to Accelerate Flow Convergence for Discontinuous Galerkin Methods in Shape Optimization DesignJournal of Scientific Computing, 2021
- Multi-label semantics preserving based deep cross-modal hashingSignal Processing: Image Communication, 2021
- Joint and individual matrix factorization hashing for large-scale cross-modal retrievalPattern Recognition, 2020
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Deep low-rank subspace ensemble for multi-view clusteringInformation Sciences, 2019
- Attention-Aware Deep Adversarial Hashing for Cross-Modal Retrieval2018
- Cross-Modal Hamming Hashing2018
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering2018
- Deep Cross-Modal Hashing2017
- Microsoft COCO: Common Objects in Context2014