Contextual Attention Network for Emotional Video Captioning
- 15 June 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 1858-1867
Abstract
This paper investigates an emerging and challenging task—emotional video captioning. Formally, given a video, the task aims to not only describe the factual content of the video, but also discover the emotional clues in the video. We propose a novel Contextual Attention Network (CANet), which recognizes and describes the fact and emotion in the video by semantic-rich context learning. To be specific, at each time step, we first extract visual and textual features from both input video and previously generated words. Then, we apply the attention mechanism to these features to capture informative contexts for captioning. We train the CANet model with the joint optimization of cross-entropy loss $\mathcal {L}_{CE}$ and contrastive loss $\mathcal {L}_{CL}$ , where $\mathcal {L}_{CE}$ constrains the semantics of the generated sentence to be close to human annotation and $\mathcal {L}_{CL}$ encourages discriminative representation learning from positive and negative pairs of video and caption. Experiments on two emotional video captioning datasets (i.e., EmVidCap and EmVidCap-S) demonstrate the superiority of CANet compared to the state-of-the-art approaches.
Keywords
Funding Information
- National Natural Science Foundation of China (61725203, 62020106007, 61876058)
- Fundamental Research Funds for the Central Universities (JZ2020HGTB0020)
This publication has 45 references indexed in Scilit:
- Sequence to Sequence -- Video to Text2015
- Describing Videos by Exploiting Temporal Structure2015
- CIDEr: Consensus-based image description evaluation2015
- Translating Videos to Natural Language Using Deep Recurrent Neural Networks2015
- Coherent Multi-sentence Video Description with Variable Level of Detail2014
- Meteor Universal: Language Specific Translation Evaluation for Any Target Language2014
- Glove: Global Vectors for Word Representation2014
- Combining modality specific deep neural networks for emotion recognition in video2013
- Translating Video Content to Natural Language Descriptions2013
- BLEU2001