Summarization of Videos with the Signature Transform
Open Access
- 5 April 2023
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 12 (7) , 1735
Abstract
This manuscript presents a new benchmark for assessing the quality of visual summaries without the need for human annotators. It is based on the Signature Transform, specifically focusing on the RMSE and the MAE Signature and Log-Signature metrics, and builds upon the assumption that uniform random sampling can offer accurate summarization capabilities. We provide a new dataset comprising videos from Youtube and their corresponding automatic audio transcriptions. Firstly, we introduce a preliminary baseline for automatic video summarization, which has at its core a Vision Transformer, an image–text model pre-trained with Contrastive Language–Image Pre-training (CLIP), as well as a module of object detection. Following that, we propose an accurate technique grounded in the harmonic components captured by the Signature Transform, which delivers compelling accuracy. The analytical measures are extensively evaluated, and we conclude that they strongly correlate with the notion of a good summary.Keywords
All Related Versions
Funding Information
- HK Innovation and Technology Commission (PID2021-122580NB-I00)
- Universitat Politècnica de València (PID2021-122580NB-I00)
- MCIN/AEI/10.13039/501100011033 and ERDF (PID2021-122580NB-I00)
- Goethe-University Frankfurt (PID2021-122580NB-I00)
- Center for Data Science & AI (PID2021-122580NB-I00)
This publication has 34 references indexed in Scilit:
- Hierarchical Recurrent Neural Network for Video Summarization2017
- Unsupervised Video Summarization with Adversarial LSTM Networks2017
- Enhancing Video Summarization via Vision-Language Embedding2017
- Video Summarization with Long Short-Term Memory2016
- Deep Residual Learning for Image Recognition2016
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- Video summarization by learning submodular mixtures of objectives2015
- Creating Summaries from User Videos2014
- VSUMM: A mechanism designed to produce static video summaries and a novel evaluation methodPattern Recognition Letters, 2011
- Long Short-Term MemoryNeural Computation, 1997