Cross-Modal Learning to Rank via Latent Joint Representation

Authors:

Abstract
Cross-modal ranking is a research topic that is imperative to many applications involving multimodal data. Discovering a joint representation for multimodal data and learning a ranking function are essential in order to boost the cross-media retrieval (i.e., image-query-text or text-query-image). In this paper, we propose an approach to discover the latent joint representation of pairs of multimodal data (e.g., pairs of an image query and a text document) via a conditional random field and structural learning in a listwise ranking manner. We call this approach cross-modal learning to rank via latent joint representation (CML 2 R). In CML 2 R, the correlations between multimodal data are captured in terms of their sharing hidden variables (e.g., topics), and a hidden-topic-driven discriminative ranking function is learned in a listwise ranking manner. The experiments show that the proposed approach achieves a good performance in cross-media retrieval and meanwhile has the capability to learn the discriminative representation of multimodal data.
Funding Information
  • National Basic Research Program of China (2012CB316400)
  • National Natural Science Foundation of China (61472353, 61402401)
  • 863 Program (2012AA012505)
  • China Knowledge Centre for Engineering Sciences and Technology
  • Fundamental Research Funds for the Central Universities
  • Zhejiang Provincial Natural Science Foundation of China (LQ14F010004, LY14F020027)
  • U.S. National Science Foundation (CCF-1017828)
  • Zhejiang Provincial Engineering Center on Media Data Cloud Processing and Analysis

This publication has 23 references indexed in Scilit: