Fangming Zhong

Currently, the semi-supervised paradigm that integrates a few labeled and large numbers of unlabeled instances has garnered significant attention in cross-modal retrieval. This approach is particularly advantageous as it not only harnesses the supervisory signals from labeled data but also capitalizes on the latent information embedded in unlabeled samples. Nevertheless, previous works predominantly focus on pairwise relationships between instances, overlooking the higher-order relationships among samples, and consequently failing to fully exploit the underlying structure of the data. To bridge this gap, we propose a novel hypergraph attention convolutional hashing (HGACH) method for semi-supervised cross-modal retrieval. Distinguished from prior works, HGACH incorporates a dedicated mechanism to prioritize unlabeled samples, enabling the model to precisely capture complex higher-order dependencies via hypergraph attention convolutional networks. This approach ensures that the interdependencies between different modalities are better understood, thus significantly boosting retrieval accuracy. In addition, a robust similarity matrix is carefully designed, explicitly modeling both inter-modality and intra-modality distances, which is essential for correctly identifying similarities between different types of data. Furthermore, for labeled instances, we propose a supervised loss to preserve the similarity according to the given labels, ensuring that the model's predictions are consistent with the labeled data. Experimental results demonstrate that the proposed HGACH method outperforms existing state-of-the-art (SOTA) methods in terms of retrieval performance, showcasing its effectiveness in handling complex cross-modal retrieval tasks. The codes are available at https://anonymous.4open.science/r/HGACH-1D7F/.

Ship image classification identifies the type of ships in an input image, which plays a significant role in the marine field. To enhance the ship classification performance, various research focuses on studying multi-modal ship classification, which aims at combining the advantages of visible images and infrared images to capture complementary information. However, the current methods simply concatenate features of different modalities to learn complementary information, which neglects the multi-level correlation between different modalities. Moreover, the existing methods require a large amount of labeled ship images to train the model. How to capture the multi-level cross-modal correlation between unlabeled and labeled data is still a challenge. In this paper, a novel semi-supervised multi-modal ship classification approach is proposed to solve these issues, which consists of two components, i.e., multi-level cross-modal interactive network and semi-supervised contrastive learning strategy. To learn comprehensive complementary information for classification, the multi-level cross-modal interactive network is designed to build local-level and global-level cross-modal feature correlation. Then, the semi-supervised contrastive learning strategy is employed to drive the optimization of the network with the intra-class consistency constraint based on supervision signals of unlabeled samples and prior label information. Extensive experiments on the public datasets demonstrate that our approach achieves state-of-the-art semi-supervised classification effectiveness.

Unsupervised cross-modal hashing retrieval across image and text modality is a challenging task because of the suboptimality of similarity guidance, i.e., the joint similarity matrix constructed by existing methods does not possess clear enough guiding significance. How to construct more robust similarity matrix is the key to solve this problem. The unsupervised cross-modal retrieval methods based on graph have a good performance in mining semantic information of input samples, but the graph hashing based on traditional affinity graph cannot capture the high-order semantic information of input samples effectively. In order to overcome the aforementioned limitations, this paper presents a novel hypergraph-based approach for unsupervised cross-modal retrieval that differs from previous works in two significant ways. Firstly, to address the ubiquitous redundant information present in current methods, this paper introduces a robust similarity matrix constructing method. Secondly, we propose a novel hypergraph enhanced module that produces embedding vectors by hypergraph convolution and attention mechanism for input data, capturing important high-order semantics. Our approach is evaluated on the NUS-WIDE and MIRFlickr datasets, and yields state-of-the-art performance for unsupervised cross-modal retrieval.