Multi-Modal Multi-Scale Deep Learning for Large-Scale Image Annotation
Authors:
- 16 November 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 28 (4) , 1720-1731
Abstract
Image annotation aims to annotate a given image with a variable number of class labels corresponding to diverse visual concepts. In this paper, we address two main issues in large-scale image annotation: 1) how to learn a rich feature representation suitable for predicting a diverse set of visual concepts ranging from object, scene to abstract concept and 2) how to annotate an image with the optimal number of class labels. To address the first issue, we propose a novel multi-scale deep model for extracting rich and discriminative features capable of representing a wide range of visual concepts. Specifically, a novel two-branch deep neural network architecture is proposed, which comprises a very deep main network branch and a companion feature fusion network branch designed for fusing the multi-scale features computed from the main branch. The deep model is also made multi-modal by taking noisy user-provided tags as model input to complement the image input. For tackling the second issue, we introduce a label quantity prediction auxiliary task to the main label prediction task to explicitly estimate the optimal label number for a given image. Extensive experiments are carried out on two large-scale image annotation benchmark datasets, and the results show that our method significantly outperforms the state of the art.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (61573363)
- Fundamental Research Funds for the Central Universities and the Research Funds of Renmin University of China (15XNLQ01)
- Renmin University of China
This publication has 32 references indexed in Scilit:
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Semantic Sparse Recoding of Visual Content for Image ApplicationsIEEE Transactions on Image Processing, 2014
- Glove: Global Vectors for Word Representation2014
- Classifier chains for multi-label classificationMachine Learning, 2011
- Large scale image annotation: learning to rank with joint word-image embeddingsMachine Learning, 2010
- Image categorization via robust pLSAPattern Recognition Letters, 2010
- LabelMe: A Database and Web-Based Tool for Image AnnotationInternational Journal of Computer Vision, 2007
- Multi-Label ClassificationInternational Journal of Data Warehousing and Mining, 2007
- Long Short-Term MemoryNeural Computation, 1997
- Invariant image recognition by Zernike momentsIEEE Transactions on Pattern Analysis and Machine Intelligence, 1990