HalluciNet-ing Spatiotemporal Representations Using a 2D-CNN
Open Access
Abstract
Spatiotemporal representations learned using 3D convolutional neural networks (CNN) are currently used in state-of-the-art approaches for action-related tasks. However, 3D-CNN are notorious for being memory and compute resource intensive as compared with more simple 2D-CNN architectures. We propose to hallucinate spatiotemporal representations from a 3D-CNN teacher with a 2D-CNN student. By requiring the 2D-CNN to predict the future and intuit upcoming activity, it is encouraged to gain a deeper understanding of actions and how they evolve. The hallucination task is treated as an auxiliary task, which can be used with any other action-related task in a multitask learning setting. Thorough experimental evaluation, it is shown that the hallucination task indeed helps improve performance on action recognition, action quality assessment, and dynamic scene recognition tasks. From a practical standpoint, being able to hallucinate spatiotemporal representations without an actual 3D-CNN can enable deployment in resource-constrained scenarios, such as with limited computing power and/or lower bandwidth. We also observed that our hallucination task has utility not only during the training phase, but also during the pre-training phase.Keywords
All Related Versions
This publication has 48 references indexed in Scilit:
- VI-Net—View-Invariant Quality of Human Movement AssessmentSensors, 2020
- Learning Effective Skeletal Representations on RGB Video for Fine-Grained Human Action Quality AssessmentElectronics, 2020
- Automated Fine Motor Evaluation for Developmental Coordination DisorderIEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019
- Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification2018
- Motion Feature Network: Fixed Motion Filter for Action Recognition2018
- Temporal Relational Reasoning in Videos2018
- End-to-End Joint Semantic Segmentation of Actors and Actions in Video2018
- An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders2016
- Anticipating Human Activities Using Object Affordances for Reactive Robotic ResponseIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015
- Long Short-Term MemoryNeural Computation, 1997