Twitter100k: A Real-world Dataset for Weakly Supervised Cross-Media Retrieval
Preprint
Abstract
This paper contributes a new large-scale dataset for weakly supervised cross-media retrieval, named Twitter100k. Current datasets, such as Wikipedia, NUS Wide and Flickr30k, have two major limitations. First, these datasets are lacking in content diversity, i.e., only some pre-defined classes are covered. Second, texts in these datasets are written in well-organized language, leading to inconsistency with realistic applications. To overcome these drawbacks, the proposed Twitter100k dataset is characterized by two aspects: 1) it has 100,000 image-text pairs randomly crawled from Twitter and thus has no constraint in the image categories; 2) text in Twitter100k is written in informal language by the users. Since strongly supervised methods leverage the class labels that may be missing in practice, this paper focuses on weakly supervised learning for cross-media retrieval, in which only text-image pairs are exploited during training. We extensively benchmark the performance of four subspace learning methods and three variants of the Correspondence AutoEncoder, along with various text features on Wikipedia, Flickr30k and Twitter100k. Novel insights are provided. As a minor contribution, inspired by the characteristic of Twitter100k, we propose an OCR-based cross-media retrieval method. In experiment, we show that the proposed OCR-based method improves the baseline performance.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Deep Multimodal Learning for Affective Analysis and RetrievalIEEE Transactions on Multimedia, 2015
- Learning Consistent Feature Representation for Cross-Modal Multimedia RetrievalIEEE Transactions on Multimedia, 2015
- Sparse Multi-Modal HashingIEEE Transactions on Multimedia, 2013
- Effective Multiple Feature Hashing for Large-Scale Near-Duplicate Video RetrievalIEEE Transactions on Multimedia, 2013
- Interactive Video Indexing With Statistical Active LearningIEEE Transactions on Multimedia, 2011
- Harmonizing Hierarchical Manifolds for Multimedia Document Semantics Understanding and Cross-Media RetrievalIEEE Transactions on Multimedia, 2008
- Mining Semantic Correlation of Heterogeneous Multimedia Data for Cross-Media RetrievalIEEE Transactions on Multimedia, 2008
- Distinctive Image Features from Scale-Invariant KeypointsInternational Journal of Computer Vision, 2004
- Modeling the Shape of the Scene: A Holistic Representation of the Spatial EnvelopeInternational Journal of Computer Vision, 2001
- The FERET evaluation methodology for face-recognition algorithmsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2000