Using Multimodal Contrastive Knowledge Distillation for Video-Text Retrieval
- 14 March 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 33 (10) , 5486-5497
Abstract
Cross-modal retrieval aims to enable a flexible bi-directional retrieval experience across different modalities ( e.g ., searching for videos with texts). Many existing efforts tend to learn a common semantic representation embedding space in which items of different modalities can be directly compared, wherein the positive global representations of video-text pairs are pulled close while the negative ones are pushed apart via pair-wise ranking loss. However, such a vanilla loss would unfortunately yield ambiguous feature embeddings for texts of different videos, causing inaccurate cross-modal matching and unreliable retrievals. Toward this end, we propose a multimodal contrastive knowledge distillation method for instance video-text retrieval, called MCKD, by adaptively using the general knowledge of self-supervised model (teacher) to calibrate mixed boundaries. Specifically, the teacher model is tailored for robust (less-ambiguous) visual-text joint semantic space by maximizing mutual information of co-occurred modalities during multimodal contrastive learning. This robust and structural inter-instance knowledge is then distilled, with the help of explicit discrimination loss, to a student model for improved matching performance. Extensive experiments on four public benchmark video-text datasets (MSR-VTT, TGIF, VATEX, and Youtube2Text) demonstrate that our MCKD can achieve at most 8.8%, 6.4%, 5.9%, and 5.3% improvement in text-to-video performance by the R@1 metric, compared with 14 SoTA baselines.Keywords
Funding Information
- science and technology innovation Program of Hunan Province (2021RC2071, 2022RC3061)
- National Natural Science Foundation of China (62072465, 62102425, 62172155, 62201014)
- Postgraduate Research and Innovation Project of Hunan Province (CX20210080)
This publication has 40 references indexed in Scilit:
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Large-Scale Video Retrieval Using Image QueriesIEEE Transactions on Circuits and Systems for Video Technology, 2017
- TGIF: A New Dataset and Benchmark on Animated GIF Description2016
- Deep Residual Learning for Image Recognition2016
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- Cross-modal Retrieval with Correspondence Autoencoder2014
- Composite Concept Discovery for Zero-Shot Video Event Detection2014
- Glove: Global Vectors for Word Representation2014
- YouTube2Text: Recognizing and Describing Arbitrary Activities Using Semantic Hierarchies and Zero-Shot Recognition2013
- Cross-media semantic representation via bi-directional learning to rank2013