AAP-MIT: Attentive Atrous Pyramid Network and Memory Incorporated Transformer for Multisentence Video Description
- 22 August 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 31 (10577149) , 5559-5569
Abstract
Generating multi-sentence descriptions for video is considered to be the most complex task in computer vision and natural language understanding due to the intricate nature of video-text data. With the recent advances in deep learning approaches, the multi-sentence video description has achieved an impressive progress. However, learning rich temporal context representation of visual sequences and modelling long-term dependencies of natural language descriptions is still a challenging problem. Towards this goal, we propose an Attentive Atrous Pyramid network and Memory Incorporated Transformer (AAP-MIT) for multi-sentence video description. The proposed AAP-MIT incorporates the effective representation of visual scene by distilling the most informative and discriminative spatio-temporal features of video data at multiple granularities and further generates the highly summarized descriptions. Profoundly, we construct AAP-MIT with three major components: i) a temporal pyramid network, which builds the temporal feature hierarchy at multiple scales by convolving the local features at temporal space, ii) a temporal correlation attention to learn the relations among various temporal video segments, and iii) the memory incorporated transformer, which augments the new memory block in language transformer to generate highly descriptive natural language sentences. Finally, the extensive experiments on ActivityNet Captions and YouCookII datasets demonstrate the substantial superiority of AAP-MIT over the existing approaches.Keywords
This publication has 42 references indexed in Scilit:
- Hierarchical Memory Modelling for Video Captioning2018
- Towards Personalized Image Captioning via Multimodal Memory NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Dense-Captioning Events in Videos2017
- Video Captioning With Attention-Based LSTM and Semantic ConsistencyIEEE Transactions on Multimedia, 2017
- Residual Attention Network for Image Classification2017
- Deep Residual Learning for Image Recognition2016
- ActivityNet: A large-scale video benchmark for human activity understanding2015
- CIDEr: Consensus-based image description evaluation2015
- Natural Language Description of Human Activities from Video Images Based on Concept Hierarchy of ActionsInternational Journal of Computer Vision, 2002
- Long Short-Term MemoryNeural Computation, 1997