Know More Say Less: Image Captioning Based on Scene Graphs
- 30 January 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 21 (8) , 2117-2130
Abstract
Automatically describing the content of an image has been attracting considerable research attention in the multimedia field. To represent the content of an image, many approaches directly utilize convolutional neural networks (CNNs) to extract visual representations, which are fed into recurrent neural networks (RNNs) to generate natural language. Recently, some approaches have detected semantic concepts from images and then encoded them into high-level representations. Although substantial progress has been achieved, most of the previous methods treat entities in images individually, thus lacking structured information that provides important cues for image captioning. In this work, we propose a framework based on scene graphs for image captioning. Scene graphs contain abundant structured information because they not only depict object entities in images but also present pairwise relationships. To leverage both visual features and semantic knowledge in structured scene graphs, we extract CNN features from the bounding box offsets of object entities for visual representations, and extract semantic relationship features from triples (e.g., man riding bike) for semantic representations . After obtaining these features, we introduce a hierarchical-attention-based module to learn discriminative features for word generation at each time step. The experimental results on benchmark datasets demonstrate the superiority of our method compared with several state-of-the-art methods.Keywords
Funding Information
- National Natural Science Foundation of China (61532018)
- Beijing Natural Science Foundation (L182054)
- National Program for Special Support of Eminent Professionals
- National Program for Support of Top-notch Young Professionals
This publication has 52 references indexed in Scilit:
- Describing Multimedia Content Using Attention-Based Encoder-Decoder NetworksIEEE Transactions on Multimedia, 2015
- Large Scale Retrieval and Generation of Image DescriptionsInternational Journal of Computer Vision, 2015
- Deep visual-semantic alignments for generating image descriptions2015
- Show and tell: A neural image caption generator2015
- CIDEr: Consensus-based image description evaluation2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Learning Consistent Feature Representation for Cross-Modal Multimedia RetrievalIEEE Transactions on Multimedia, 2015
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- On the Properties of Neural Machine Translation: Encoder–Decoder Approaches2014
- BLEU2001