Joint Video Summarization and Moment Localization by Cross-Task Sample Transfer
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 16367-16377
Abstract
Video summarization has recently engaged increasing attention in computer vision communities. However, the scarcity of annotated data has been a key obstacle in this task. To address it, this work explores a new solution for video summarization by transferring samples from a correlated task (i.e., video moment localization) equipped with abundant training data. Our main insight is that the annotated video moments also indicate the semantic highlights of a video, essentially similar to video summary. Approximately, the video summary can be treated as a sparse, redundancy-free version of the video moments. Inspired by this observation, we propose an importance Propagation based collaborative Teaching Network (iPTNet). It consists of two separate modules that conduct video summarization and moment localization, respectively. Each module estimates a frame-wise importance map for indicating keyframes or moments. To perform cross-task sample transfer, we devise an importance propagation module that realizes the conversion between summarization-guided and localization-guided importance maps. This way critically enables optimizing one of the tasks using the data from the other task. Additionally, in order to avoid error amplification caused by batch-wise joint training, we devise a collaborative teaching scheme, which adopts a crosstask mean teaching strategy to realize the joint optimization of the two tasks and provide robust frame-level teaching signals. Extensive experiments on video summarization benchmarks demonstrate that iPTNet significantly outperforms previous state-of-the-art video summarization methods, serving as an effective solution that overcomes the data scarcity issue in video summarization.Keywords
Funding Information
- National Key R&D Program of China (2020AAA0104401)
- Beijing Natural Science Foundation (Z190001)
This publication has 78 references indexed in Scilit:
- Query-Focused Extractive Video Summarization2016
- Highlight Detection with Pairwise Deep Ranking for First-Person Video Summarization2016
- Summary Transfer: Exemplar-Based Subset Selection for Video Summarization2016
- Video summarization by learning submodular mixtures of objectives2015
- Going deeper with convolutions2015
- Video co-summarization: Video summarization by visual co-occurrence2015
- A Bag-of-Importance Model With Locality-Constrained Coding Based Feature Learning for Video SummarizationIEEE Transactions on Multimedia, 2014
- Glove: Global Vectors for Word Representation2014
- Story-Driven Summarization for Egocentric Video2013
- VSUMM: A mechanism designed to produce static video summaries and a novel evaluation methodPattern Recognition Letters, 2011