Language-Guided Graph Representation Learning for Video Summarization
- 17 November 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (3) , 3216-3232
Abstract
With the rapid growth of video content on social media, video summarization has become a crucial task in multimedia processing. However, existing methods face challenges in capturing global dependencies in video content and accommodating multimodal user customization. Moreover, temporal proximity between video frames does not always correspond to semantic proximity. To tackle these challenges, we propose a novel Language-guided Graph Representation Learning Network (LGRLN) for video summarization. Specifically, we introduce a video graph generator that converts video frames into a structured graph to preserve temporal order and contextual dependencies. By constructing forward, backward and undirected graphs, the video graph generator effectively preserves the sequentiality and contextual relationships of video content. We designed an intra-graph relational reasoning module with a dual-threshold graph convolution mechanism, which distinguishes semantically relevant frames from irrelevant ones between nodes. Additionally, our proposed language-guided cross-modal embedding module generates video summaries with specific textual descriptions. We model the summary generation output as a mixture of Bernoulli distribution and solve it with the EM algorithm. Experimental results show that our method outperforms existing approaches across multiple benchmarks. Moreover, we proposed LGRLN reduces inference time and model parameters by 87.8% and 91.7%, respectively.Keywords
All Related Versions
Funding Information
- National Key R&D Program of China (2023YFA1008501)
- National Natural Science Foundation of China (624B2049)
- National Natural Science Foundation of China (U22B2035)
- National Natural Science Foundation of China (62425101)
- National Natural Science Foundation of China (62502116)
- Shenzhen KQTD (20240729102051063)
- China Post-Doctoral Science Foundation (2025M774315)
This publication has 58 references indexed in Scilit:
- Query Focused Video Summarization: A Review2022
- SumGraph: Video Summarization via Recursive Graph Modeling2020
- Retrospective Encoders for Video Summarization2018
- Video Summarization Using Fully Convolutional Sequence Networks2018
- Query-Focused Extractive Video Summarization2016
- Video Summarization with Long Short-Term Memory2016
- Category-Specific Video Summarization2014
- Creating Summaries from User Videos2014
- The EM Algorithm and Extensions, 2E2008
- Keyframe-based video summarization using Delaunay clusteringInternational Journal on Digital Libraries, 2006