Learning and Using the Arrow of Time
Authors:
- Donglai WeiDWDonglai Wei
- Joseph LimJLJoseph Lim
- Andrew ZissermanAZAndrew Zisserman
- William T FreemanWFWilliam T Freeman
- 1 June 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 8052-8060
Abstract
We seek to understand the arrow of time in videos - what makes videos look like they are playing forwards or backwards? Can we visualize the cues? Can the arrow of time be a supervisory signal useful for activity analysis? To this end, we build three large-scale video datasets and apply a learning-based approach to these tasks. To learn the arrow of time efficiently and reliably, we design a ConvNet suitable for extended temporal footprints and for class activation visualization, and study the effect of artificial cues, such as cinematographic conventions, on learning. Our trained model achieves state-of-the-art performance on large-scale real-world video datasets. Through cluster analysis and localization of important regions for the prediction, we examine learned visual cues that are consistent among many samples and show when and where they occur. Lastly, we use the trained ConvNet for two applications: self-supervision for action recognition, and video forensics - determining whether Hollywood film clips have been deliberately reversed in time, often used as special effects.Keywords
This publication has 11 references indexed in Scilit:
- Self-Supervised Video Representation Learning with Odd-One-Out Networks2017
- Dynamic Image Networks for Action Recognition2016
- Learning Deep Features for Discriminative Localization2016
- Convolutional Two-Stream Network Fusion for Video Action Recognition2016
- YFCC100MCommunications of the ACM, 2016
- Learning Temporal Embeddings for Complex Video Analysis2015
- Unsupervised Visual Representation Learning by Context Prediction2015
- Modeling video evolution for action recognition2015
- Going deeper with convolutions2015
- Seeing the Arrow of Time2014