Dense Captioning with Joint Inference and Visual Context
Authors:
- Linjie YangLYLinjie Yang
- Kevin TangKTKevin Tang
- Jianchao YangJYJianchao Yang
- Li-Jia LiLLLi-Jia Li
- 1 July 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636919,p. 1978-1987
Abstract
Dense captioning is a newly emerging computer vision topic for understanding images with dense language descriptions. The goal is to densely detect visual concepts (e.g., objects, object parts, and interactions between them) from images, labeling each with a short descriptive phrase. We identify two key challenges of dense captioning that need to be properly addressed when tackling the problem. First, dense visual concept annotations in each image are associated with highly overlapping target regions, making accurate localization of each visual concept challenging. Second, the large amount of visual concepts makes it hard to recognize each of them by appearance alone. We propose a new model pipeline based on two novel ideas, joint inference and context fusion, to alleviate these two challenges. We design our model architecture in a methodical manner and thoroughly evaluate the variations in architecture. Our final model, compact and efficient, achieves state-of-the-art accuracy on Visual Genome [23] for dense captioning with a relative gain of 73% compared to the previous best algorithm. Qualitative experiments also reveal the semantic capabilities of our model in dense captioning.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- Deep visual-semantic alignments for generating image descriptions2015
- Show and tell: A neural image caption generator2015
- From captions to visual concepts and back2015
- Fully convolutional networks for semantic segmentation2015
- CIDEr: Consensus-based image description evaluation2015
- The Role of Context for Object Detection and Semantic Segmentation in the Wild2014
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Long Short-Term MemoryNeural Computation, 1997
- Generalization of backpropagation with application to a recurrent gas market modelNeural Networks, 1988