Video Segmentation and Tokenization for Model-Based Video Scene Classification
- 7 August 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 27 (15209210) , 6489-6502
Abstract
In this paper, we propose a novel approach for segmenting and tokenizing a video scene recording into a sequence of cascade units, known as visual segment units and modeled with visual segment models (VSMs) for video scene classification (VSC). Specifically, the proposed VSM framework takes deep visual features extracted from pre-trained encoders as inputs and models the temporal interactions between segment units by hidden Markov models. Next, we use unit co-occurrence statistics to introduce relationships between VSM units within a video scene recording. Furthermore, the VSM approach is extended to an acoustic-visual variant, subsequently integrating itself into a deep learning-based multi-modal scene classification system. This combination serves to further exploit the complementary nature of audio and video data. By incorporating a set of visual segment units into modeling a video scene class, it captures both inter-class similarity and intra-class diversity, facilitating improved scene classification, especially within categories prone to confusion. Extensive experimental results on a benchmark published by the DCASE (Detection and Classification of Acoustic Scenes and Events) 2021 Challenge show that the proposed framework can effectively handle the confusion issue among similar video scenes. In addition, our multi-modal integration system achieves state-of-the-art performance in the audio-visual scene classification task in the DCASE 2021 Challenge, thereby demonstrating the effectiveness of our proposed approach.Keywords
Funding Information
- National Natural Science Foundation of China (62401533)
- National Natural Science Foundation of China (62171427)
This publication has 59 references indexed in Scilit:
- Multimodal emotion recognition using cross modal audio-video fusion with attention and deep metric learningImage and Vision Computing, 2023
- Deep Audio-visual Learning: A SurveyInternational Journal of Automation and Computing, 2021
- Cross-Task Transfer for Geotagged Audiovisual Aerial Scene Recognition2020
- Scene recognition: A comprehensive surveyPattern Recognition, 2020
- Learning Discriminative and Shareable Features for Scene Classification2014
- Randomized Spatial Partition for Scene Recognition2012
- Gaussian Mixture Models2009
- Distinctive Image Features from Scale-Invariant KeypointsInternational Journal of Computer Vision, 2004
- Modeling the Shape of the Scene: A Holistic Representation of the Spatial EnvelopeInternational Journal of Computer Vision, 2001
- Text categorization with Support Vector Machines: Learning with many relevant features1998