Abstract
Video summarization (VS) aims to generate a representative summary of original video content, thereby promoting research in social behavior prediction and analysis. Most existing approaches analyze social behaviors and thematic content in a video from a purely visual perspective, failing to account for cross-modal thematic consistency. Furthermore, these methods suffer from several limitations in capturing both local and holistic perception and modeling long-range dependencies. Meanwhile, we are motivated to explore the intrinsic thematic consistency across different modalities. We observe a part-to-whole relationship, where the semantics of individual frames constitute the global semantic summary of a video. To quantify this relationship, we design a frame-level semantic-aligned attention (FSA) mechanism inspired by CLIP-style contrastive learning. Based on this, we propose an unsupervised VS method termed the semantic-guided hybrid-attention transformer (SG-HAT). Building upon the SUM-GAN-AAE framework, we replace the traditional LSTM-based architecture with an encoder–decoder Transformer to capture thematic consistency. Specifically, we incorporate FSA into the decoder, enabling both local and holistic perception by measuring the similarity between frame-level and global features. In addition, multihead cross-attention enables effective cross-modal interaction. By adding FSA into the frame selector, we construct a semantically driven GAN training strategy to achieve precise video reconstruction. Furthermore, with the help of vision–language models, we introduce textual information by generating dense captions and text summaries for each video in the TVSum and SumMe datasets. Experimental results on the multimodal summarization task (video-to-video, V2V) and conventional VS benchmarks demonstrate that our proposed SG-HAT achieves competitive performance.