CCAH: a CLIP-based Cycle Alignment Hashing method for unsupervised vision-text retrieval
Preprint
- 19 December 2022
- preprint
- Published by Springer Nature in Research Square
Abstract
Due to the advantages of low storage cost and fast retrieval efficiency, deep hashing methods are widely used in cross-modal retrieval. Images are usually accompanied by corresponding text descriptions rather than labels. Therefore, unsupervised methods have been widely concerned. However, due to the modal divide and semantic differences, existing unsupervised methods cannot adequately bridge the modal differences, leading to suboptimal retrieval results. In this paper, we propose CLIPbased Cycle Alignment Hashing for unsupervised vision-text retrieval (CCAH), which aims to exploit the semantic link between the original features of modalities and the reconstructed features. Firstly we design a modal cyclic interaction method that aligns semantically within intramodality, where one modal feature reconstructs another modal feature, thus taking full account of the semantic similarity between intra-modal and inter-modal relationships. Secondly, introducing GAT into crossmodal retrieval tasks. Consider the influence of text neighbour nodes and add attention mechanisms to capture the global features of text modalities. Thirdly, Fine-grained extraction of image features using the CLIP visual coder. Finally, hash encoding is learning through hash functions. The experiments demonstrate on three widely used datasets that our proposed CCAH achieves satisfactory results in total retrieval accuracy. Our code can be found at :https://github.com/CQYIO/CCAH.git.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- High-order nonlocal Hashing for unsupervised cross-modal retrievalWorld Wide Web, 2021
- Joint and individual matrix factorization hashing for large-scale cross-modal retrievalPattern Recognition, 2020
- Multi-Pathway Generative Adversarial Hashing for Unsupervised Cross-Modal RetrievalIEEE Transactions on Multimedia, 2019
- Discrete Latent Factor Model for Cross-Modal HashingIEEE Transactions on Image Processing, 2019
- Deep Ordinal Hashing With Spatial AttentionIEEE Transactions on Image Processing, 2018
- Deep Binary Reconstruction for Cross-Modal HashingIEEE Transactions on Multimedia, 2018
- Triplet-Based Deep Hashing Network for Cross-Modal RetrievalIEEE Transactions on Image Processing, 2018
- Multimodal Discriminative Binary Embedding for Large-Scale Cross-Modal RetrievalIEEE Transactions on Image Processing, 2016
- Instance-Aware Hashing for Multi-Label Image RetrievalIEEE Transactions on Image Processing, 2016
- Microsoft COCO: Common Objects in Context2014