Emotion Expression With Fact Transfer for Video Description
- 10 February 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 24 (15209210) , 715-727
Abstract
Translating a video into natural language is a fundamental but challenging task in visual understanding, since there is a great gap between visual content and linguistic sentence. More attention has been paid to this research field and a number of state-of-the-art results are achieved in recent years. However, the emotions in videos are usually overlooked, leading to the generated description sentences being boring and colorless. In this work, we construct a new dataset for video description with emotion expression, which consists of two parts: a re-annotated subset of the MSVD dataset with emotion embedded and another subset annotated with long sentences and rich emotions based on a video emotion recognition dataset. A fact transfer based framework is designed, which incorporates a fact stream and an emotion stream to generate sentences with emotion expression for video description. In addition, we propose a novel approach for sentence evaluation by balancing facts and emotions. A group of experiments are conducted, and the experimental results demonstrate the effectiveness of the proposed methods, including the idea of dataset construction for video description with emotion expression, model training and testing, and the emotion evaluation metric. The project page (including the code and dataset) can be found in https://mic.tongji.edu.cn/ce/70/c9778a183920/page.htm .Keywords
Funding Information
- National Natural Science Foundation of China (61976159, 62062041, 61962003)
- Shanghai Innovation Action Project of Science and Technology (20511100700)
- Ph.D. Research Initiation Project of Jinggangshan University (JZB1923)
- Shanghai Engineering Research Center of Industrial Vision Perception and Intelligent Computing (17DZ2251600)
This publication has 44 references indexed in Scilit:
- Describing Multimedia Content Using Attention-Based Encoder-Decoder NetworksIEEE Transactions on Multimedia, 2015
- Long-term recurrent convolutional networks for visual recognition and description2015
- Beyond short snippets: Deep networks for video classification2015
- Deep visual-semantic alignments for generating image descriptions2015
- Show and tell: A neural image caption generator2015
- CIDEr: Consensus-based image description evaluation2015
- Translating Videos to Natural Language Using Deep Recurrent Neural Networks2015
- Caffe2014
- BLEU2001
- Emotion: A Psychoevolutionary SynthesisThe American Journal of Psychology, 1980