DASZL: Dynamic Action Signatures for Zero-shot Learning
Preprint
Abstract
There are many realistic applications of activity recognition where the set of potential activity descriptions is combinatorially large. This makes end-to-end supervised training of a recognition system impractical as no training set is practically able to encompass the entire label set. In this paper, we present an approach to fine-grained recognition that models activities as compositions of dynamic action signatures. This compositional approach allows us to reframe fine-grained recognition as zero-shot activity recognition, where a detector is composed "on the fly" from simple first-principles state machines supported by deep-learned components. We evaluate our method on the Olympic Sports and UCF101 datasets, where our model establishes a new state of the art under multiple experimental paradigms. We also extend this method to form a unique framework for zero-shot joint segmentation and classification of activities in video and demonstrate the first results in zero-shot decoding of complex action sequences on a widely-used surgical dataset. Lastly, we show that we can use off-the-shelf object detectors to recognize activities in completely de-novo settings with no additional training.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Generalized zero-shot learning for action recognition with web-scale video dataWorld Wide Web, 2018
- Cross-Modal and Hierarchical Modeling of Video and Text2018
- Feature Generating Networks for Zero-Shot Learning2018
- Temporal Convolutional Networks for Action Segmentation and Detection2017
- Zero-Shot Visual Recognition via Bidirectional Latent EmbeddingInternational Journal of Computer Vision, 2017
- Transductive Zero-Shot Action Recognition by Word-Vector EmbeddingInternational Journal of Computer Vision, 2017
- A large-scale benchmark dataset for event recognition in surveillance video2011
- Recognizing human actions by attributes2011
- Semantic event representation and recognition using syntactic attribute graph grammarPattern Recognition Letters, 2009
- Recognition of visual activities and interactions by stochastic parsingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2000