GAIM: Graph Attention Interaction Model for Collective Activity Recognition
- 23 July 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 22 (2) , 524-539
Abstract
Unbalanced interaction relationships at personal and group levels play a pivotal role in collective activity recognition, which have not been adaptively and jointly explored by previous approaches. In this work, we propose a graph attention interaction model (GAIM) embedded with the graph attention block (GAB) to explicitly and adaptively infer unbalanced interaction relations at personal and group levels in a unified architecture, and further to learn the spatial and temporal evolutions of the collective activity from these interactions to predict the activity labels. We first design the spatio-temporal graphs tailored to the collective activity where the concurrent person and group nodes respectively represent individuals' actions and the collective activity. The graphs provide both spatial structures and semantic appearance features for the collective activity. Then GAB performs convolution-like filters on the graphs to infer unequal and two-level interaction relations in the collective activity by implementing graph convolutional networks (GCNs) with a shared attention mechanism. At the personal level, the GAB learns different levels of interactions for each person node from its neighbor person nodes under the guidance from the group node. At the group level, the GAB assesses various degrees of interactions to the group node contributed by person nodes. Equipped with the GRUs network, the GAIM learns the spatial and temporal evolutions of individuals' actions as well as the collective activity from the captured interactions, and finally predicts the label of the collective activity. Experiments on four publicly available datasets and ablation studies are conducted to evaluate the performance of our GAIM, and the improved performance demonstrates the effectiveness of our model.Keywords
Funding Information
- National Natural Science Foundation of China (61273273)
- National Key Research and Development (2017YFC0112001)
This publication has 33 references indexed in Scilit:
- Long-Term Recurrent Convolutional Networks for Visual Recognition and DescriptionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Look and Think Twice: Capturing Top-Down Visual Attention with Feedback Convolutional Neural Networks2015
- Visual recognition by counting instances: A multi-instance cardinality potential kernel2015
- Learning Person–Person Interaction in Collective Activity RecognitionIEEE Transactions on Image Processing, 2015
- Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks2015
- Understanding Collective Activitiesof People from VideosIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- On the Properties of Neural Machine Translation: Encoder–Decoder Approaches2014
- Context-Aware Modeling and Recognition of Activities in Video2013
- Discriminative Latent Models for Recognizing Contextual Group ActivitiesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2011
- Clothes search in consumer photos via color matching and attribute learning2011