Causal Video Summarizer for Video Exploration
Preprint
Abstract
Recently, video summarization has been proposed as a method to help video exploration. However, traditional video summarization models only generate a fixed video summary which is usually independent of user-specific needs and hence limits the effectiveness of video exploration. Multi-modal video summarization is one of the approaches utilized to address this issue. Multi-modal video summarization has a video input and a text-based query input. Hence, effective modeling of the interaction between a video input and text-based query is essential to multi-modal video summarization. In this work, a new causality-based method named Causal Video Summarizer (CVS) is proposed to effectively capture the interactive information between the video and query to tackle the task of multi-modal video summarization. The proposed method consists of a probabilistic encoder and a probabilistic decoder. Based on the evaluation of the existing multi-modal video summarization dataset, experimental results show that the proposed approach is effective with the increase of +5.4% in accuracy and +4.92% increase of F 1- score, compared with the state-of-the-art method.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Counterfactual Vision and Language Learning2020
- Auto-classification of Retinal Diseases in the Limit of Sparse Data Using a Two-Streams Machine Learning Model2019
- Synthesizing New Retinal Symptom Images by Multiple Generative Models2019
- Dual Attention Network for Scene Segmentation2019
- Identifying causal effects with proxy variables of an unmeasured confounderBiometrika, 2018
- Deep Residual Learning for Image Recognition2016
- Measurement bias and effect restoration in causal inferenceBiometrika, 2014
- Category-Specific Video Summarization2014
- Adjusting for Errors in Classification and Measurement in the Analysis of Partly and Purely Categorical DataJournal of the American Statistical Association, 1986
- Correcting for Misclassification in Two-Way Tables and Matched-Pair StudiesInternational Journal of Epidemiology, 1983