SHE-Net: Syntax-Hierarchy-Enhanced Text-Video Retrieval
- 20 February 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (8) , 7631-7641
Abstract
The user base of short video apps has experienced unprecedented growth in recent years, resulting in a significant demand for video content analysis. In particular, text-video retrieval, which aims to find the top matching videos given text descriptions from a vast video corpus, is an essential function, the primary challenge of which is to bridge the modality gap. Nevertheless, most existing approaches treat texts merely as discrete tokens and neglect their syntax structures. Moreover, the abundant spatial and temporal clues in videos are often under-utilized due to the lack of interaction with text. To address these issues, we argue that using texts as guidance to focus on relevant temporal frames and spatial regions within videos is beneficial. In this paper, we propose a novel Syntax-Hierarchy-Enhanced text-video retrieval method (SHE-Net) that exploits the inherent semantic and syntax hierarchy of texts to bridge the modality gap from two perspectives. First, to facilitate a more fine-grained integration of visual content, we employ the text syntax hierarchy, which reveals the grammatical structure of text descriptions, to guide the visual representations. Second, to further enhance the multi-modal interaction and alignment, we also utilize the syntax hierarchy to guide the similarity calculation. We evaluated our method on four public text-video retrieval datasets of MSR-VTT, MSVD, DiDeMo, and ActivityNet. The experimental results and ablation studies confirm the advantages of our proposed method.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- Revisiting Temporal Modeling for CLIP-Based Image-to-Video Knowledge Transferring2023
- Deep learning for video-text retrieval: a reviewInternational Journal of Multimedia Information Retrieval, 2023
- TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- X-Pool: Cross-Modal Language-Video Attention for Text-Video Retrieval2022
- Multi-modal Transformer for Video Retrieval2020
- Fine-Grained Video-Text Retrieval With Hierarchical Graph Reasoning2020
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- ActivityNet: A large-scale video benchmark for human activity understanding2015