Implicit and Explicit Concept Relations in Deep Neural Networks for Multi-Label Video/Image Annotation
Open Access
- 18 June 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 29 (6) , 1631-1644
Abstract
In this work we propose a DCNN (Deep Convolutional Neural Network) architecture that addresses the problem of video/image concept annotation by exploiting concept relations at two different levels. At the first level, we build on ideas from multi-task learning, and propose an approach to learn conceptspecific representations that are sparse, linear combinations of representations of latent concepts. By enforcing the sharing of the latent concept representations, we exploit the implicit relations between the target concepts. At a second level, we build on ideas from structured output learning, and propose the introduction, at training time, of a new cost term that explicitly models the correlations between the concepts. By doing so, we explicitly model the structure in the output space (i.e., the concept labels). Both of the above are implemented using standard convolutional layers and are incorporated in a single DCNN architecture that can then be trained end-to-end with standard back-propagation. Experiments on four large-scale video and image datasets show that the proposed DCNN improves concept annotation accuracy and outperforms the related state-of-the-art methods.Keywords
Funding Information
- H2020 LEIT Information and Communication Technologies (H2020-687786 InVID)
- Nvidia
This publication has 42 references indexed in Scilit:
- Joint Structural Learning to Rank with Deep Linear Feature LearningIEEE Transactions on Knowledge and Data Engineering, 2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Local Features and a Two-Layer Stacking Architecture for Semantic Concept Detection in VideoIEEE Transactions on Emerging Topics in Computing, 2015
- Multi-label classification with Bayesian network-based chain classifiersPattern Recognition Letters, 2014
- Image Annotation by Multiple-Instance Learning With Discriminative Feature Mapping and SelectionIEEE Transactions on Cybernetics, 2013
- Semantic context learning with large-scale weakly-labeled image set2012
- Adaptive spatial information-theoretic clustering for image segmentationPattern Recognition, 2009
- NUS-WIDE2009
- Uncovering deterministic causal structures: a Boolean approachSynthese, 2008
- Convex multi-task feature learningMachine Learning, 2008