Zero-shot Recognition of Complex Action Sequences
Preprint
Abstract
Zero-shot video classification for fine-grained activity recognition has largely been explored using methods similar to its image-based counterpart, namely by defining image-derived attributes that serve to discriminate among classes. However, such methods do not capture the fundamental dynamics of activities and are thus limited to cases where static image content alone suffices to classify an activity. For example, reversible actions such as entering and exiting a car are often indistinguishable. In this work, we present a framework for straightforward modeling of activities as a state machine of dynamic attributes. We show that encoding the temporal structure of attributes greatly increases our modeling power, allowing us to capture action direction, for example. Further, we can extend this to activity detection using dynamic programming, providing, to our knowledge, the first example of zero-shot joint segmentation and classification of complex action sequences in a larger video. We evaluate our method on the Olympic Sports dataset where our model establishes a new state of the art for standard zero-shot-learning (ZSL) evaluation as well as outperforming all other models in the inductive category for general (GZSL) zero-shot evaluation. Additionally, we are the first to demonstrate zero-shot decoding of complex action sequences on a widely used surgical dataset. Lastly, we show that that we can even eliminate the need to train attribute detectors by using off-the-shelf object detectors to recognize activities in challenging surveillance videos.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Generalized zero-shot learning for action recognition with web-scale video dataWorld Wide Web, 2018
- Temporal Relational Reasoning in Videos2018
- Cross-Modal and Hierarchical Modeling of Video and Text2018
- Temporal Convolutional Networks for Action Segmentation and Detection2017
- Zero-Shot Visual Recognition via Bidirectional Latent EmbeddingInternational Journal of Computer Vision, 2017
- Transductive Zero-Shot Action Recognition by Word-Vector EmbeddingInternational Journal of Computer Vision, 2017
- Objects2action: Classifying and Localizing Actions without Any Video Example2015
- Unsupervised Domain Adaptation for Zero-Shot Learning2015
- A large-scale benchmark dataset for event recognition in surveillance video2011
- Weighted Automata Algorithms2009