Global-Shared Text Representation Based Multi-Stage Fusion Transformer Network for Multi-Modal Dense Video Captioning
- 23 August 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 26 (15209210) , 3164-3179
Abstract
Dense video captioning aims to detect all events of an uncropped video and generate corresponding textual captions for each event. Multi-modal information is essential to improve the performance of this task, but the existing methods mainly rely on the single visual or dual audio-visual modal input, while completely ignoring the text modal input (subtitle). Since the text data has a similar data representation as video caption words, it is conducive to the performance improvement of video captioning. In this paper, we propose a novel framework, called the multi-stage fusion transformer network (MS-FTN), to realize multi-modal dense video captioning by fusing the text, the audio, and the visual features in stages. We present a multi-stage feature fusion encoder that first fuses audio and visual modalities at a lower level and then fuses them with a global-shared text representation at a higher level to generate a set of multi-modal complementary context features. In addition, an anchor-free event proposal module is proposed to efficiently generate a set of event proposals without the complex anchor calculation. Extensive experiments on the subsets of the ActivityNet Captions dataset show that our proposed MS-FTN achieves superior performance and efficient computation. Moreover, the ablation studies demonstrate that the global-shared text representation is more suitable for multi-modal dense video captioning. Our code and data are available at https://github.com/xieyulai/GS-MS-FTN .Keywords
Funding Information
- Fundamental Research Funds for the Central University, China (FRF-BD-20-11A)
This publication has 52 references indexed in Scilit:
- Multimodal sentiment analysis based on fusion methods: A surveyInformation Fusion, 2023
- Multi-modal Transformer for Video Retrieval2020
- A Fine-Grained Spatial-Temporal Attention Model for Video CaptioningIEEE Access, 2018
- Move Forward and Tell: A Progressive Generator of Video Descriptions2018
- A Joint Sequence Fusion Model for Video Question Answering and Retrieval2018
- Multimodal architecture for video captioning with memory networks and an attention mechanismPattern Recognition Letters, 2018
- Early and late integration of audio features for automatic video description2017
- Long-term recurrent convolutional networks for visual recognition and description2015
- Coherent Multi-sentence Video Description with Variable Level of Detail2014
- Visualizing and Understanding Convolutional Networks2014