Com-STAL: Compositional Spatio-Temporal Action Localization
- 17 May 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 33 (12) , 7645-7657
Abstract
Spatio-temporal action localization aims to locate the spatial and temporal positions of actors and classify their actions. However, prior research overlooks the fact that human actions often interact with novel objects in real-world scenarios, which neglects the various combinations of action-object, and considerably limits the generalization of the developed models. In this paper, we study the action-object combinations by researching multi-modal vision information of them. To this end, we propose a novel compositional spatio-temporal action localization (Com-STAL) task, which features non-overlapping action-object combinations in their training and test sets. Based on this, we construct a compositional action localization dataset (Com-AD). Beyond that, we propose a simple yet effective framework, Instance-Centric Interaction Network (ICIN), to reduce invalid induction biases within the visual modality and alleviate the combined distribution bias issue by leveraging additional modal information. The extensive experiment results on Com-AD demonstrate superior action localization performance of ICIN.Keywords
Funding Information
- National Key Research and Development Program of China (2018AAA0102001)
- National Natural Science Foundation of China (62222207, 62072245, 61932020)
- Natural Science Foundation of Jiangsu Province (BK20211520)
This publication has 70 references indexed in Scilit:
- ContextLocNet: Context-Aware Deep Network Models for Weakly Supervised Localization2016
- Temporal Action Localization with Pyramid of Score Distribution Features2016
- Sequence to Sequence -- Video to Text2015
- Describing Videos by Exploiting Temporal Structure2015
- DevNet: A Deep Event Network for multimedia event detection and evidence recounting2015
- Towards Understanding Action Recognition2013
- A large-scale benchmark dataset for event recognition in surveillance video2011
- A Survey on Visual Surveillance of Object Motion and BehaviorsIEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 2004
- A New Approach to Linear Filtering and Prediction ProblemsJournal of Basic Engineering, 1960
- The Hungarian method for the assignment problemNaval Research Logistics Quarterly, 1955