Elaborative Rehearsal for Zero-shot Action Recognition
Preprint
Abstract
The growing number of action classes has posed a new challenge for video understanding, making Zero-Shot Action Recognition (ZSAR) a thriving direction. The ZSAR task aims to recognize target (unseen) actions without training examples by leveraging semantic representations to bridge seen and unseen actions. However, due to the complexity and diversity of actions, it remains challenging to semantically represent action classes and transfer knowledge from seen data. In this work, we propose an ER-enhanced ZSAR model inspired by an effective human memory technique Elaborative Rehearsal (ER), which involves elaborating a new concept and relating it to known concepts. Specifically, we expand each action class as an Elaborative Description (ED) sentence, which is more discriminative than a class name and less costly than manual-defined attributes. Besides directly aligning class semantics with videos, we incorporate objects from the video as Elaborative Concepts (EC) to improve video semantics and generalization from seen actions to unseen actions. Our ER-enhanced ZSAR model achieves state-of-the-art results on three existing benchmarks. Moreover, we propose a new ZSAR evaluation protocol on the Kinetics dataset to overcome limitations of current benchmarks and demonstrate the first case where ZSAR performance is comparable to few-shot learning baselines on this more realistic setting. We will release our codes and collected EDs at https://github.com/DeLightCMU/ElaborativeRehearsal.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- ZEST: Zero-shot Learning from Text Descriptions using Textual Similarity and Visual Summarization2020
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Temporal Segment Networks for Action Recognition in VideosIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the UglyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Transductive Zero-Shot Action Recognition by Word-Vector EmbeddingInternational Journal of Computer Vision, 2017