Encouraging LSTMs to Anticipate Actions Very Early
Preprint
Abstract
In contrast to the widely studied problem of recognizing an action given a complete sequence, action anticipation aims to identify the action from only partially available videos. As such, it is therefore key to the success of computer vision applications requiring to react as early as possible, such as autonomous navigation. In this paper, we propose a new action anticipation method that achieves high prediction accuracy even in the presence of a very small percentage of a video sequence. To this end, we develop a multi-stage LSTM architecture that leverages context- and action-aware features, and introduce a novel loss function that encourages the model to predict the correct class as early as possible. Our experiments on standard benchmark datasets evidence the benefits of our approach; We outperform the state-of-the-art action anticipation methods for early prediction by a relative increase in accuracy of 22.0% on JHMDB-21, 14.0% on UT-Interaction and 49.9% on UCF-101.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Online Localization and Prediction of Actions and InteractionsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Going deeper into action recognition: A surveyImage and Vision Computing, 2017
- A Multi-stream Bi-directional Recurrent Neural Network for Fine-Grained Action Detection2016
- Real-Time Action Recognition with Enhanced Motion Vector CNNs2016
- DeepCAMP: Deep Convolutional Action & Attribute Mid-Level Patterns2016
- Convolutional Two-Stream Network Fusion for Video Action Recognition2016
- VLAD3: Encoding Dynamics of Deep Features for Action Recognition2016
- Actionness Estimation Using Hybrid Fully Convolutional Networks2016
- Rank Pooling for Action RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Middle-Level Representation for Human Activities Recognition: The Role of Spatio-Temporal Relationships2012