Learning cross-modality similarity for multinomial data
- 1 November 2011
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15505499,p. 2407-2414
Abstract
Many applications involve multiple-modalities such as text and images that describe the problem of interest. In order to leverage the information present in all the modalities, one must model the relationships between them. While some techniques have been proposed to tackle this problem, they either are restricted to words describing visual objects only, or require full correspondences between the different modalities. As a consequence, they are unable to tackle more realistic scenarios where a narrative text is only loosely related to an image, and where only a few image-text pairs are available. In this paper, we propose a model that addresses both these challenges. Our model can be seen as a Markov random field of topic models, which connects the documents based on their similarity. As a consequence, the topics learned with our model are shared across connected documents, thus encoding the relations between different modalities. We demonstrate the effectiveness of our model for image retrieval from a loosely related text.Keywords
This publication has 11 references indexed in Scilit:
- A new approach to cross-modal multimedia retrieval2010
- Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora2010
- Topic regression multi-modal Latent Dirichlet Allocation for image annotation2010
- Hierarchical Dirichlet ProcessesJournal of the American Statistical Association, 2006
- Pachinko allocation2006
- Finding scientific topicsProceedings of the National Academy of Sciences, 2004
- Video Google: a text retrieval approach to object matching in videos2003
- Modeling annotated data2003
- 10.1162/jmlr.2003.3.4-5.993Applied Physics Letters, 2000
- 10.1162/153244303322533214Applied Physics Letters, 2000