Group Activity Recognition by Using Effective Multiple Modality Relation Representation With Temporal-Spatial Attention
Open Access
- 10 March 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 8, 65689-65698
Abstract
Group activity recognition has received a great deal of interest because of its broader applications in sports analysis, autonomous vehicles, CCTV surveillance systems and video summarization systems. Most existing methods typically use appearance features and they seldom consider underlying interaction information. In this work, a technology of novel group activity recognition is proposed based on multi-modal relation representation with temporal-spatial attention. First, we introduce an object relation module, which processes all objects in a scene simultaneously through an interaction between their appearance feature and geometry, thus allowing the modeling of their relations. Second, to extract effective motion features, an optical flow network is fine-tuned by using the action loss as the supervised signal. Then, we propose two types of inference models, opt-GRU and relation-GRU, which are used to encode the object relationship and motion representation effectively, and form the discriminative frame-level feature representation. Finally, an attention-based temporal aggregation layer is proposed to integrate frame-level features with different weights and form effective video-level representations. We have performed extensive experiments on two popular datasets, and both have achieved state-of-the-art performance. The datasets are the Volleyball dataset and the Collective Activity dataset, respectively.Keywords
Funding Information
- National Natural Science Foundation of China (61976010, 61702022, 6180211)
- Beijing Municipal Education Committee Science Foundation (KM201910005024)
- Beijing Excellent Young Talent Cultivation Project (2017000020124G075)
- Beijing Postdoctoral Research Foundation (ZZ2019-63)
- Beijing University of Technology Ri Xin Cultivation Project
This publication has 34 references indexed in Scilit:
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition2016
- Convolutional Two-Stream Network Fusion for Video Action Recognition2016
- Detecting Events and Key Actors in Multi-person Videos2016
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- DevNet: A Deep Event Network for multimedia event detection and evidence recounting2015
- Visual recognition by counting instances: A multi-instance cardinality potential kernel2015
- On the Properties of Neural Machine Translation: Encoder–Decoder Approaches2014
- Supervised Sequence Labelling2012
- A Database and Evaluation Methodology for Optical FlowInternational Journal of Computer Vision, 2010
- Beyond Tracking: Modelling Activity and Understanding BehaviourInternational Journal of Computer Vision, 2006