Detecting Events and Key Actors in Multi-person Videos
Authors:
- Vignesh RamanathanVRVignesh Ramanathan
- Jonathan HuangJHJonathan Huang
- Sami Abu-El-HaijaSASami Abu-El-Haija
- Alexander GorbanAGAlexander Gorban
- Kevin MurphyKMKevin Murphy
- Li Fei-FeiLFLi Fei-Fei
- 1 June 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3043-3053
Abstract
Multi-person event recognition is a challenging task, often with many people active in the scene but only a small subset contributing to an actual event. In this paper, we propose a model which learns to detect events in such videos while automatically "attending" to the people responsible for the event. Our model does not use explicit annotations regarding who or where those people are during training and testing. In particular, we track people in videos and use a recurrent neural network (RNN) to represent the track features. We learn time-varying attention weights to combine these features at each time-instant. The attended features are then processed using another RNN for event detection/ classification. Since most video datasets with multiple people are restricted to a small number of videos, we also collected a new basketball dataset comprising 257 basketball games with 14K event annotations corresponding to 11 event classes. Our model outperforms state-of-the-art methods for both event classification and detection on this new dataset. Additionally, we show that the attention mechanism is able to consistently localize the relevant players.Keywords
All Related Versions
This publication has 43 references indexed in Scilit:
- Multimedia event detection with multimodal feature fusion and temporal concept localizationMachine Vision and Applications, 2013
- Spatiotemporal Deformable Part Models for Action Detection2013
- Better Exploiting Motion for Better Action Recognition2013
- Learning to Track and Identify Players from Broadcast Sports VideosIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Evaluation of local spatio-temporal features for action recognition2009
- Detecting group activities using rigidity of formation2005
- Recognizing Planned, Multiperson ActionComputer Vision and Image Understanding, 2001
- Resolving motion correspondence for densely moving pointsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2001
- A model of saliency-based visual attention for rapid scene analysisIEEE Transactions on Pattern Analysis and Machine Intelligence, 1998
- Algorithms for the Assignment and Transportation ProblemsJournal of the Society for Industrial and Applied Mathematics, 1957