General Knowledge Embedded Image Representation Learning
Authors:
- 11 July 2017
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 20 (1) , 198-207
Abstract
Image representation learning is a fundamental problem in understanding semantics of images. However, traditional classification-based representation learning methods face the noisy and incomplete problem of the supervisory labels. In this paper, we propose a general knowledge base embedded image representation learning approach, which uses general knowledge graph, which is a multitype relational knowledge graph consisting of human commonsense beyond image space, as external semantic resource to capture the relations of concepts in image representation learning. A relational regularized regression CNN (R 3 CNN) model is designed to jointly optimize the image representation learning problem and knowledge graph embedding problem. In this manner, the learnt representation can capture not only labeled tags but also related concepts of images, which involves more precise and complete semantics. Comprehensive experiments are conducted to investigate the effectiveness and transferability of our approach in tag prediction task, zero-shot tag inference task, and content-based image retrieval task. The experimental results demonstrate that the proposed approach performs significantly better than the existing representation learning methods. Finally, observation of the learnt relations show that our approach can somehow refine the knowledge base to describe images and label the images with structured tags.Keywords
Funding Information
- National Program on Key Basic Research Project (2015CB352300)
- National Natural Science Foundation of China (U1611461, 61521002, 61531006, 61210008)
This publication has 23 references indexed in Scilit:
- Zero-shot Image Tagging by Hierarchical Semantic Embedding2015
- Learning Compact Hash Codes for Multimodal Representations Using Orthogonal Deep StructureIEEE Transactions on Multimedia, 2015
- Enhancing Video Event Recognition Using Automatically Constructed Semantic-Visual Knowledge BaseIEEE Transactions on Multimedia, 2015
- Traversing Knowledge Graphs in Vector Space2015
- CNN Features Off-the-Shelf: An Astounding Baseline for Recognition2014
- Linked tag: image annotation using semantic relationships between image tagsMultimedia Tools and Applications, 2014
- Picture tags and world knowledge2013
- All About VLAD2013
- NUS-WIDE2009
- Hamming Embedding and Weak Geometric Consistency for Large Scale Image Search2008