Bidirectional Cross-Modal Collaborative Alignment via Semantic-Guided Visual Embeddings for Partially Relevant Video Retrieval
- 2 February 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 35 (10577149) , 1423-1435
Abstract
Partially Relevant Video Retrieval (PRVR) aims to retrieve videos that match a given textual query only partially. This task is inherently challenging due to the modality gap between text and video, which is further exacerbated by the partial semantic correspondence between linguistic descriptions and visual content. To address these challenges, we propose a bidirectional cross-modal alignment mechanism that collaboratively optimizes both visual and textual modalities. In the visual modality, a major difficulty lies in the absence of visual cues that directly correspond to textual semantics, limiting the models ability to align visual representations with textual meanings under unsupervised conditions. To overcome this issue, we construct a semantic-visual association library, which stores paired visual and textual features with semantic annotations. During training, the model dynamically retrieves the most semantically similar visual samples from this library based on the current visual feature vector. These retrieved samples, preliminarily associated with semantics via cross-modal matching, are used to form dynamic anchors that guide visual representation learning. By leveraging these enriched visual features, the model progressively refines the visual representations to achieve better alignment with the corresponding textual inputs, thereby enhancing cross-modal consistency. In the textual modality, we enhance textual representations by integrating semantically aligned visual features selected from the same association library, further narrowing the modality gap. Extensive experiments on benchmark datasets under partial semantic correspondence scenarios demonstrate that our method achieves state-of-the-art performance. The source code of the paper is available at https://github.com/cyanlll/BOA.Keywords
Funding Information
- National Natural Science Foundation of China (62276120)
- Yunnan Fundamental Research Projects (202301AV070004)
- Yunnan Fundamental Research Projects (202401AS070640)
This publication has 40 references indexed in Scilit:
- TeachText: CrossModal text-video retrieval through generalized distillationArtificial Intelligence, 2024
- Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?2023
- VoP: Text-Video Co-Operative Prompt Tuning for Cross-Modal Retrieval2023
- Deep learning for video-text retrieval: a reviewInternational Journal of Multimedia Information Retrieval, 2023
- EclipSE: Efficient Long-Range Video Retrieval Using Sight and Sound2022
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval2020
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Deep Residual Learning for Image Recognition2016