Early Embedding and Late Reranking for Video Captioning
Authors:
Open Access
- 1 October 2016
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1082-1086
Abstract
This paper describes our solution for the MSR Video to Language Challenge. We start from the popular ConvNet + LSTM model, which we extend with two novel modules. One is early embedding, which enriches the current low-level input to LSTM by tag embeddings. The other is late reranking, for re-scoring generated sentences in terms of their relevance to a specific video. The modules are inspired by recent works on image captioning, repurposed and redesigned for video. As experiments on the MSR-VTT validation set show, the joint use of these two modules add a clear improvement over a non-trivial ConvNet + LSTM baseline under four performance metrics. The viability of the proposed solution is further confirmed by the blind test by the organizers. Our system is ranked at the 4th place in terms of overall performance, while scoring the best CIDEr-D, which measures the human-likeness of generated captions.Keywords
Funding Information
- National Natural Science Foundation of China (61303184)
- the Fundamental Research Funds for the Central Universities and the Research Funds of Renmin University of China (14XNLQ01)
- National Social Science Fundation of China (12&ZD141)
This publication has 14 references indexed in Scilit:
- Improving Image Captioning by Concept-Based Sentence Reranking2016
- Socializing the Semantic GapACM Computing Surveys, 2016
- The ImageNet Shuffle2016
- Sequence to Sequence -- Video to Text2015
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- Image Tag Assignment, Refinement and Retrieval2015
- Going deeper with convolutions2015
- Show and tell: A neural image caption generator2015
- From captions to visual concepts and back2015
- CIDEr: Consensus-based image description evaluation2015