Actor and Action Video Segmentation from a Sentence
Authors:
- Kirill GavrilyukKGKirill Gavrilyuk
- Amir GhodratiAGAmir Ghodrati
- Zhenyang LiZLZhenyang Li
- Cees G. M. SnoekCSCees G. M. Snoek
- 1 June 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 5958-5966
Abstract
This paper strives for pixel-level segmentation of actors and their actions in video content. Different from existing works, which all learn to segment from a fixed vocabulary of actor and action pairs, we infer the segmentation from a natural language input sentence. This allows to distinguish between fine-grained actors in the same super-category, identify actor and action instances, and segment pairs that are outside of the actor and action vocabulary. We propose a fully-convolutional model for pixel-level actor and action segmentation using an encoder-decoder architecture optimized for video. To show the potential of actor and action video segmentation from a sentence, we extend two popular actor and action datasets with more than 7,500 natural language descriptions. Experiments demonstrate the quality of the sentence-guided segmentations, the generalization ability of our model, and its advantage for traditional actor and action segmentation compared to the state-of-the-art.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Actor-Action Semantic Segmentation with Grouping Process Models2016
- Natural Language Object Retrieval2016
- Generation and Comprehension of Unambiguous Object Descriptions2016
- Objects2action: Classifying and Localizing Actions without Any Video Example2015
- ActivityNet: A large-scale video benchmark for human activity understanding2015
- Can humans fly? Action understanding with multiple classes of actors2015
- Fast action proposals for human action detection and search2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Towards Understanding Action Recognition2013