Learning from Collective Intelligence
Authors:
Open Access
- 2 November 2016
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 13 (1) , 1-23
Abstract
Feature representation for visual content is the key to the progress of many fundamental applications such as annotation and cross-modal retrieval. Although recent advances in deep feature learning offer a promising route towards these tasks, they are limited in application domains where high-quality and large-scale training data are expensive to obtain. In this article, we propose a novel deep feature learning paradigm based on social collective intelligence, which can be acquired from the inexhaustible social multimedia content on the Web, in particular, largely social images and tags. Differing from existing feature learning approaches that rely on high-quality image-label supervision, our weak supervision is acquired by mining the visual-semantic embeddings from noisy, sparse, and diverse social image collections. The resultant image-word embedding space can be used to (1) fine-tune deep visual models for low-level feature extractions and (2) seek sparse representations as high-level cross-modal features for both image and text. We offer an easy-to-use implementation for the proposed paradigm, which is fast and compatible with any state-of-the-art deep architectures. Extensive experiments on several benchmarks demonstrate that the cross-modal features learned by our paradigm significantly outperforms others in various applications such as content-based retrieval, classification, and image captioning.Keywords
Funding Information
- NUS-Tsinghua Extreme Search (NExT) project (R-252-300-001-490)
This publication has 41 references indexed in Scilit:
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Robust (Semi) Nonnegative Graph EmbeddingIEEE Transactions on Image Processing, 2014
- A Unified Framework of Latent Feature Learning in Social MediaIEEE Transactions on Multimedia, 2014
- Improving Image-Sentence Embeddings Using Large Weakly Annotated Photo Collections2014
- Clickage2013
- Right buddy makes the difference2012
- Efficient Object Category Recognition Using Classemes2010
- NUS-WIDE2009
- Smooth Optimization with Approximate GradientSIAM Journal on Optimization, 2008
- Indexing by latent semantic analysisJournal of the American Society for Information Science, 1990