Weakly Supervised Dense Video Captioning
Authors:
- Zhiqiang ShenZSZhiqiang Shen
- Jianguo LiJLJianguo Li
- Zhou SuZSZhou Su
- Minjun LiMLMinjun Li
- Yurong ChenYCYurong Chen
- Yu-Gang JiangYJYu-Gang Jiang
- Xiangyang XueXXXiangyang Xue
- 1 July 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 10636919,p. 5159-5167
Abstract
This paper focuses on a novel and challenging vision task, dense video captioning, which aims to automatically describe a video clip with multiple informative and diverse caption sentences. The proposed method is trained without explicit annotation of fine-grained sentence to video region-sequence correspondence, but is only based on weak video-level sentence annotations. It differs from existing video captioning systems in three technical aspects. First, we propose lexical fully convolutional neural networks (Lexical-FCN) with weakly supervised multi-instance multi-label learning to weakly link video regions with lexical labels. Second, we introduce a novel submodular maximization scheme to generate multiple informative and diverse region-sequences based on the Lexical-FCN outputs. A winner-takes-all scheme is adopted to weakly associate sentences to region-sequences in the training phase. Third, a sequence-to-sequence learning based language model is trained with the weakly supervised information obtained through the association process. We show that the proposed method can not only produce informative and diverse dense captions, but also outperform state-of-the-art single video captioning methods by a large margin.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- Multimodal Video Description2016
- Jointly Modeling Embedding and Translation to Bridge Video and Language2016
- You Only Look Once: Unified, Real-Time Object Detection2016
- Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning2016
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Coherent Multi-sentence Video Description with Variable Level of Detail2014
- Softening quantization in bag-of-audio-words2014
- Cost-effective outbreak detection in networks2007
- An analysis of approximations for maximizing submodular set functions—IMathematical Programming, 1978