Cross-Modality Retrieval by Joint Correlation Learning
Open Access
- 30 April 2019
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 15 (2s) , 1-16
Abstract
As an indispensable process of cross-media analyzing, comprehending heterogeneous data faces challenges in the fields of visual question answering (VQA), visual captioning, and cross-modality retrieval. Bridging the semantic gap between the two modalities is still difficult. In this article, to address the problem in cross-modality retrieval, we propose a cross-modal learning model with joint correlative calculation learning. First, an auto-encoder is used to embed the visual features by minimizing the error of feature reconstruction and a multi-layer perceptron (MLP) is utilized to model the textual features embedding. Then we design a joint loss function to optimize both the intra- and the inter-correlations among the image-sentence pairs, i.e., the reconstruction loss of visual features, the relevant similarity loss of paired samples, and the triplet relation loss between positive and negative examples. In the proposed method, we optimize the joint loss based on a batch score matrix and utilize all mutual mismatched paired samples to enhance its performance. Our experiments in the retrieval tasks demonstrate the effectiveness of the proposed method. It achieves comparable performance to the state-of-the-art on three benchmarks, i.e., Flickr8k, Flickr30k, and MS-COCO.Keywords
Funding Information
- National Natural Science Foundation of China (61725203, 61732008, 61876058)
- National Key Research and Development Program of China (2018YFB0804200)
This publication has 36 references indexed in Scilit:
- Harnessing Semantics for Answer Sentence Retrieval2015
- Show and tell: A neural image caption generator2015
- Deep correlation for matching images and text2015
- From captions to visual concepts and back2015
- Associating neural word embeddings with deep image representations using Fisher Vectors2015
- Cross-modal Retrieval with Correspondence Autoencoder2014
- Image Retrieval Using Textual Cues2013
- Histograms of Oriented Gradients for Human Detection2005
- Artificial neural networks (the multilayer perceptron)—a review of applications in the atmospheric sciencesAtmospheric Environment, 1998
- The expectation-maximization algorithmIEEE Signal Processing Magazine, 1996