MoLo: Motion-Augmented Long-Short Contrastive Learning for Few-Shot Action Recognition
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 18011-18021
Abstract
Current state-of-the-art approaches for few-shot action recognition achieve promising performance by conducting frame-level matching on learned visual features. However, they generally suffer from two limitations: i) the matching procedure between local frames tends to be inaccurate due to the lack of guidance to force long-range temporal perception; ii) explicit motion learning is usually ignored, leading to partial information loss. To address these issues, we develop a Motion-augmented Long-short Contrastive Learning (MoLo) method that contains two crucial components, including a long-short contrastive objective and a motion autodecoder. Specifically, the long-short contrastive objective is to endow local frame features with long-form temporal awareness by maximizing their agreement with the global token of videos belonging to the same class. The motion autodecoder is a lightweight architecture to reconstruct pixel motions from the differential features, which explicitly embeds the network with motion dynamics. By this means, MoLo can simultaneously learn long-range temporal context and motion cues for comprehensive few-shot matching. To demonstrate the effectiveness, we evaluate MoLo on five standard benchmarks, and the results show that MoLo favorably outperforms recent advanced methods. The source code is available at https://github.com/alibaba-mmai-research/MoLo.Keywords
Funding Information
- National Natural Science Foundation of China (U22B2053)
This publication has 59 references indexed in Scilit:
- A Robust and Efficient Video Representation for Action RecognitionInternational Journal of Computer Vision, 2015
- Better Exploiting Motion for Better Action Recognition2013
- Vision-Based Finger Action Recognition by Angle Detection and Contour AnalysisETRI Journal, 2011
- One Shot Similarity Metric Learning for Action Recognition2011
- A survey on vision-based human action recognitionImage and Vision Computing, 2010
- Modeling Temporal Structure of Decomposable Motion Segments for Activity Classification2010
- Dynamic Time Warping2007
- Action Recognition in Broadcast Tennis Video Using Optical Flow and Support Vector Machine2006
- The computation of optical flowACM Computing Surveys, 1995
- Determining optical flowArtificial Intelligence, 1981