UBiSS: A Unified Framework for Bimodal Semantic Summarization of Videos
Open Access
- 30 May 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1034-1042
Abstract
With the surge in the amount of video data, video summarization techniques, including visual-modal(VM) and textual-modal(TM) summarization, are attracting more and more attention. However, unimodal summarization inevitably loses the rich semantics of the video. In this paper, we focus on a more comprehensive video summarization task named Bimodal Semantic Summarization of Videos(BiSSV). Specifically, we first construct a large-scale dataset, BIDS, in(video, VM-Summary, TM-Summary) triplet format. Unlike traditional processing methods, our construction procedure contains a VM-Summary extraction algorithm aiming to preserve the most salient content within long videos. Based on BIDS, we propose a Unified framework UBiSS for the BiSSV task, which models the saliency information in the video and generates a TM-summary and VM-summary simultaneously. We further optimize our model with a list-wise ranking-based objective to improve its capacity to capture highlights. Lastly, we propose a metric, NDCGMS, to provide a joint evaluation of the bimodal summary. Experiments show that our unified framework achieves better performance than multi-stage summarization pipelines. Code and data are available at https://github.com/MeiYutingg/UBiSS.Keywords
All Related Versions
Funding Information
- Beijing Natural Science Foundation (L233008)
- National Natural Science Foundation of China (62072462)
This publication has 32 references indexed in Scilit:
- TL;DW? Summarizing Instructional Videos with Task Relevance and Cross-Modal Saliency2022
- Deep attentive and semantic preserving video summarizationNeurocomputing, 2020
- Summarizing Videos with Attention2019
- Rethinking the Evaluation of Video Summaries2019
- Video Summarization with Long Short-Term Memory2016
- Highlight Detection with Pairwise Deep Ranking for First-Person Video Summarization2016
- Going deeper with convolutions2015
- Video co-summarization: Video summarization by visual co-occurrence2015
- CIDEr: Consensus-based image description evaluation2015
- Creating Summaries from User Videos2014