Rich Image Captioning in the Wild
Authors:
- Kenneth TranKTKenneth Tran
- Xiaodong HeXHXiaodong He
- Lei ZhangLZLei Zhang
- Jian SunJSJian Sun
- 1 June 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 434-441
Abstract
We present an image caption system that addresses new challenges of automatically describing images in the wild. The challenges include generating high quality caption with respect to human judgments, out-of-domain data handling, and low latency required in many applications. Built on top of a state-of-the-art framework, we developed a deep vision model that detects a broad range of visual concepts, an entity recognition model that identifies celebrities and landmarks, and a confidence model for the caption output. Experimental results show that our caption engine outperforms previous state-of-the-art systems significantly on both in-domain dataset (i.e. MS COCO) and out-of-domain datasets. We also make the system publicly accessible as a part of the Microsoft Cognitive Services.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- MS-Celeb-1M: Challenge of Recognizing One Million Celebrities in the Real WorldElectronic Imaging, 2016
- Long-term recurrent convolutional networks for visual recognition and description2015
- Deep visual-semantic alignments for generating image descriptions2015
- Show and tell: A neural image caption generator2015
- From captions to visual concepts and back2015
- CIDEr: Consensus-based image description evaluation2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- A Latent Semantic Model with Convolutional-Pooling Structure for Information Retrieval2014
- Baby talk: Understanding and generating simple image descriptions2011
- Freebase2008