Learning Visual N-Grams from Web Data
Authors:
- Ang LiALAng Li
- Allan JabriAJAllan Jabri
- Armand JoulinAJArmand Joulin
- Laurens van der MaatenLMLaurens van der Maaten
- 1 October 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4193-4202
Abstract
Real-world image recognition systems need to recognize tens of thousands of classes that constitute a plethora of visual concepts. The traditional approach of annotating thousands of images per class for training is infeasible in such a scenario, prompting the use of webly supervised data. This paper explores the training of image-recognition systems on large numbers of images and associated user comments, without using manually labeled images. In particular, we develop visual n-gram models that can predict arbitrary phrases that are relevant to the content of an image. Our visual n-gram models are feed-forward convolutional networks trained using new loss functions that are inspired by n-gram models commonly used in language modeling. We demonstrate the merits of our models in phrase prediction, phrase-based image retrieval, relating images and captions, and zero-shot transfer.Keywords
All Related Versions
This publication has 45 references indexed in Scilit:
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Visual Turing test for computer vision systemsProceedings of the National Academy of Sciences, 2015
- A Simple Method to Determine if a Music Information Retrieval System is a “Horse”IEEE Transactions on Multimedia, 2014
- Bilingually-constrained Phrase Embeddings for Machine Translation2014
- Every Picture Tells a Story: Generating Sentences from Images2010
- Efficient Object Category Recognition Using Classemes2010
- OPTIMOL: Automatic Online Picture Collection via Incremental Model LearningInternational Journal of Computer Vision, 2009
- A bit of progress in language modelingComputer Speech & Language, 2001
- Content-based image retrieval at the end of the early yearsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2000
- Estimation of probabilities from sparse data for the language model component of a speech recognizerIEEE Transactions on Acoustics, Speech, and Signal Processing, 1987