Consistency Prototype Module and Motion Compensation for Few-Shot Action Recognition (CLIP-CP$\mathbf{M^2}$C)
Preprint
Abstract
Recently, few-shot action recognition has significantly progressed by learning the feature discriminability and designing suitable comparison methods. Still, there are the following restrictions. (a) Previous works are mainly based on visual mono-modal. Although some multi-modal works use labels as supplementary to construct prototypes of support videos, they can not use this information for query videos. The labels are not used efficiently. (b) Most of the works ignore the motion feature of video, although the motion features are essential for distinguishing. We proposed a Consistency Prototype and Motion Compensation Network(CLIP-CP$M^2$C) to address these issues. Firstly, we use the CLIP for multi-modal few-shot action recognition with the text-image comparison for domain adaption. Secondly, in order to make the amount of information between the prototype and the query more similar, we propose a novel method to compensate for the text(prompt) information of query videos when text(prompt) does not exist, which depends on a Consistency Loss. Thirdly, we use the differential features of the adjacent frames in two directions as the motion features, which explicitly embeds the network with motion dynamics. We also apply the Consistency Loss to the motion features. Extensive experiments on standard benchmark datasets demonstrate that the proposed method can compete with state-of-the-art results. Our code is available at the URL: https://github.com/xxx/xxx.git.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Multimodal Prototype-Enhanced Network for Few-Shot Action Recognition2024
- Hierarchical compositional representations for few-shot action recognitionComputer Vision and Image Understanding, 2024
- CLIP-guided Prototype Modulating for Few-shot Action RecognitionInternational Journal of Computer Vision, 2023
- MoLo: Motion-Augmented Long-Short Contrastive Learning for Few-Shot Action Recognition2023
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- VDARN: Video Disentangling Attentive Relation Network for Few-Shot and Zero-Shot Action RecognitionAd Hoc Networks, 2021
- Few-shot activity recognition with cross-modal memory networkPattern Recognition, 2020
- Label Independent Memory for Semi-Supervised Few-shot Video ClassificationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Information Retrieval for Music and Motion2007