The Pose Knows: Video Forecasting by Generating Pose Futures
Authors:
- Jacob WalkerJWJacob Walker
- Kenneth MarinoKMKenneth Marino
- Abhinav GuptaAGAbhinav Gupta
- Martial HebertMHMartial Hebert
Top Cited Papers
- 1 October 2017
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3352-3361
Abstract
Current approaches to video forecasting attempt to generate videos directly in pixel space using Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs). However, since these approaches try to model all the structure and scene dynamics at once, in unconstrained settings they often generate uninterpretable results. Our insight is that forecasting needs to be done first at a higher level of abstraction. Specifically, we exploit human pose detectors as a free source of supervision and break the video forecasting problem into two discrete steps. First we explicitly model the high level structure of active objects in the scene (humans) and use a VAE to model the possible future movements of humans in the pose space. We then use the future poses generated as conditional information to a GAN to predict the future frames of the video in pixel space. By using the structured space of pose as an intermediate representation, we sidestep the problems that GANs have in generating video pixels directly. We show through quantitative and qualitative evaluation that our method outperforms state-of-the-art methods for video prediction.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Learning Social Etiquette: Human Trajectory Understanding In Crowded Scenes2016
- Context Encoders: Feature Learning by Inpainting2016
- Learning Activity Progression in LSTMs for Activity Detection and Early Detection2016
- Convolutional Pose Machines2016
- Dense Optical Flow Prediction from a Static Image2015
- Max-Margin Early Event DetectorsInternational Journal of Computer Vision, 2013
- Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural EnvironmentsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- From Actemes to Action: A Strongly-Supervised Representation for Detailed Action Understanding2013
- HMDB: A large video database for human motion recognition2011
- A Data-Driven Approach for Event Prediction2010