Video Captioning Based on Channel Soft Attention and Semantic Reconstructor
Open Access
- 23 February 2021
- journal article
- research article
- Published by MDPI AG in Future Internet
- Vol. 13 (2) , 55
Abstract
Video captioning is a popular task which automatically generates a natural-language sentence to describe video content. Previous video captioning works mainly use the encoder–decoder framework and exploit special techniques such as attention mechanisms to improve the quality of generated sentences. In addition, most attention mechanisms focus on global features and spatial features. However, global features are usually fully connected features. Recurrent convolution networks (RCNs) receive 3-dimensional features as input at each time step, but the temporal structure of each channel at each time step has been ignored, which provide temporal relation information of each channel. In this paper, a video captioning model based on channel soft attention and semantic reconstructor is proposed, which considers the global information for each channel. In a video feature map sequence, the same channel of every time step is generated by the same convolutional kernel. We selectively collect the features generated by each convolutional kernel and then input the weighted sum of each channel to RCN at each time step to encode video representation. Furthermore, a semantic reconstructor is proposed to rebuild semantic vectors to ensure the integrity of semantic information in the training process, which takes advantage of both forward (semantic to sentence) and backward (sentence to semantic) flows. Experimental results on popular datasets MSVD and MSR-VTT demonstrate the effectiveness and feasibility of our model.Keywords
Funding Information
- National Natural Science Foundation of China (No. 61572306)
- National Key Research and Development Program of China (NO. 2017YFB0701600)
- Science and Technology Commission of Shanghai Municipality (No. 19511121002)
This publication has 41 references indexed in Scilit:
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Effective Approaches to Attention-based Neural Machine Translation2015
- Translating Videos to Natural Language Using Deep Recurrent Neural Networks2015
- Large-Scale Video Classification with Convolutional Neural Networks2014
- Translating Video Content to Natural Language Descriptions2013
- Action Recognition with Improved Trajectories2013
- Image Classification with the Fisher Vector: Theory and PracticeInternational Journal of Computer Vision, 2013
- Domain Based Ontology and Automated Text Categorization Based on Improved Term Frequency – Inverse Document FrequencyInternational Journal of Modern Education and Computer Science, 2012
- Action recognition by dense trajectories2011
- BLEU2001