On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval
Top Cited Papers
- 2 August 2013
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 36 (3) , 521-535
Abstract
The problem of cross-modal retrieval from multimedia repositories is considered. This problem addresses the design of retrieval systems that support queries across content modalities, for example, using an image to search for texts. A mathematical formulation is proposed, equating the design of cross-modal retrieval systems to that of isomorphic feature spaces for different content modalities. Two hypotheses are then investigated regarding the fundamental attributes of these spaces. The first is that low-level cross-modal correlations should be accounted for. The second is that the space should enable semantic abstraction. Three new solutions to the cross-modal retrieval problem are then derived from these hypotheses: correlation matching (CM), an unsupervised method which models cross-modal correlations, semantic matching (SM), a supervised technique that relies on semantic representation, and semantic correlation matching (SCM), which combines both. An extensive evaluation of retrieval performance is conducted to test the validity of the hypotheses. All approaches are shown successful for text retrieval in response to image queries and vice versa. It is concluded that both hypotheses hold, in a complementary form, although evidence in favor of the abstraction hypothesis is stronger than that for correlation.Keywords
This publication has 53 references indexed in Scilit:
- On the regularization of image semantics by modal expansion2012
- Introduction to Information Retrieval2008
- Semantic Annotation and Retrieval of Music and Sound EffectsIEEE Transactions on Audio, Speech, and Language Processing, 2008
- Supervised Learning of Semantic Classes for Image Annotation and RetrievalIEEE Transactions on Pattern Analysis and Machine Intelligence, 2007
- Generalized histogram intersection kernel for image recognition2005
- Canonical Correlation Analysis: An Overview with Application to Learning MethodsNeural Computation, 2004
- Musical genre classification of audio signalsIEEE Transactions on Speech and Audio Processing, 2002
- 10.1162/153244303322533214Applied Physics Letters, 2000
- Color indexingInternational Journal of Computer Vision, 1991
- Relations Between Two Sets of VariatesBiometrika, 1936