Long-Short Temporal Contrastive Learning of Video Transformers
Authors:
Preprint
Abstract
Video transformers have recently emerged as a competitive alternative to 3D CNNs for video understanding. However, due to their large number of parameters and reduced inductive biases, these models require supervised pretraining on large-scale image datasets to achieve top performance. In this paper, we empirically demonstrate that self-supervised pretraining of video transformers on video-only datasets can lead to action recognition results that are on par or better than those obtained with supervised pretraining on large-scale image datasets, even massive ones such as ImageNet-21K. Since transformer-based models are effective at capturing dependencies over extended temporal spans, we propose a simple learning procedure that forces the model to match a long-term view to a short-term view of the same video. Our approach, named Long-Short Temporal Contrastive Learning (LSTCL), enables video transformers to learn an effective clip-level representation by predicting temporal context captured from a longer temporal extent. To demonstrate the generality of our findings, we implement and validate our approach under three different self-supervised contrastive learning frameworks (MoCo v3, BYOL, SimSiam) using two distinct video-transformer architectures, including an improved variant of the Swin Transformer augmented with space-time attention. We conduct a thorough ablation study and show that LSTCL achieves competitive performance on multiple video benchmarks and represents a convincing alternative to supervised image-based pretraining.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- An Empirical Study of Training Self-Supervised Vision Transformers2021
- Emerging Properties in Self-Supervised Vision Transformers2021
- Broaden Your Views for Self-Supervised Video Learning2021
- Exploring Simple Siamese Representation Learning2021
- End-to-End Video Instance Segmentation with Transformers2021
- A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning2021
- Spatiotemporal Contrastive Video Representation Learning2021
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Self-Supervised Video Hashing With Hierarchical Binary Auto-EncoderIEEE Transactions on Image Processing, 2018