Fine-Grained Spatiotemporal Motion Alignment for Contrastive Video Representation Learning
Open Access
- 26 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 4725-4736
Abstract
As the most essential property in a video, motion information is critical to a robust and generalized video representation. To inject motion dynamics, recent works have adopted frame difference as the source of motion information in video contrastive learning, considering the trade-off between quality and cost. However, existing works align motion features at the instance level, which suffers from spatial and temporal weak alignment across modalities. In this paper, we present a Fine-grained Motion Alignment (FIMA) framework, capable of introducing well-aligned and significant motion information. Specifically, we first develop a dense contrastive learning framework in the spatiotemporal domain to generate pixel-level motion supervision. Then, we design a motion decoder and a foreground sampling strategy to eliminate the weak alignments in terms of time and space. Moreover, a frame-level motion contrastive loss is presented to improve the temporal diversity of the motion features. Extensive experiments demonstrate that the representations learned by FIMA possess great motion-awareness capabilities and achieve state-of-the-art or competitive results on downstream tasks across UCF101, HMDB51, and Diving48 datasets. Code is available at https://github.com/ZMHH-H/FIMA.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62233013)
- National Natural Science Foundation of China (62173248)
- National Natural Science Foundation of China (62073245)
- Suzhou Key Industry Technological Innovation-Core Technology R&D Program (SGC2021035)
This publication has 48 references indexed in Scilit:
- Self-Supervised Video Representation Learning with Space-Time Cubic PuzzlesProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Self-Supervised Spatio-Temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics2019
- RESOUND: Towards Action Recognition Without Representation Bias2018
- Learning and Using the Arrow of Time2018
- Unsupervised Feature Learning via Non-parametric Instance Discrimination2018
- A Closer Look at Spatiotemporal Convolutions for Action Recognition2018
- Non-local Neural Networks2018
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification2016
- Learning Deep Features for Discriminative Localization2016