On the regularization of image semantics by modal expansion
- 1 June 2012
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636919,p. 3093-3099
Abstract
Recent research efforts in semantic representations and context modeling are based on the principle of task expansion: that vision problems such as object recognition, scene classification, or retrieval (RCR) cannot be solved in isolation. The extended principle of modality expansion (that RCR problems cannot be solved from visual information alone) is investigated in this work. A semantic image labeling system is augmented with text. Pairs of images and text are mapped to a semantic space, and the text features used to regularize their image counterparts. This is done with a new cross-modal regularizer, which learns the mapping of the image features that maximizes their average similarity to those derived from text. The proposed regularizer is class-sensitive, combining a set of class-specific denoising transformations and nearest neighbor interpolation of text-based class assignments. Regularization of a state-of-the-art approach to image retrieval is then shown to produce substantial gains in retrieval accuracy, outperforming recent image retrieval approaches.Keywords
This publication has 20 references indexed in Scilit:
- Towards cross-category knowledge propagation for learning visual concepts2011
- Towards semantic knowledge propagation from text corpus to web images2011
- A new approach to cross-modal multimedia retrieval2010
- Introduction to Information Retrieval2008
- A study of query by semantic example2008
- Putting Objects in PerspectiveInternational Journal of Computer Vision, 2008
- Bridging the Gap: Query by Semantic ExampleIEEE Transactions on Multimedia, 2007
- Objects in Context2007
- Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories2006
- Distinctive Image Features from Scale-Invariant KeypointsInternational Journal of Computer Vision, 2004