A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3298-3308
Abstract
We present a large-scale study on unsupervised spatiotemporal representation learning from videos. With a unified perspective on four recent image-based frameworks, we study a simple objective that can easily generalize all these methods to space-time. Our objective encourages temporally-persistent features in the same video, and in spite of its simplicity, it works surprisingly well across: (i) different unsupervised frameworks, (ii) pre-training datasets, (iii) downstream datasets, and (iv) backbone architectures. We draw a series of intriguing observations from this study, e.g., we discover that encouraging long-spanned persistency can be effective even if the timespan is 60 seconds. In addition to state-of-the-art results in multiple benchmarks, we report a few promising cases in which unsupervised pre-training can outperform its supervised counterpart. Code will be made available at https://github.com/facebookresearch/SlowFast.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- Deep Residual Learning for Image Recognition2016
- Visually Indicated Sounds2016
- Learning to See by Moving2015
- Unsupervised Visual Representation Learning by Context Prediction2015
- Unsupervised Learning of Spatiotemporally Coherent Metrics2015
- Learning Image Representations Tied to Ego-Motion2015
- Unsupervised Learning of Visual Representations Using Videos2015
- Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Going deeper with convolutions2015
- Slow Feature Analysis: Unsupervised Learning of InvariancesNeural Computation, 2002