Semantic-Disentangled Transformer With Noun-Verb Embedding for Compositional Action Recognition
- 15 December 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 297-309
Abstract
Recognizing actions performed on unseen objects, known as Compositional Action Recognition (CAR), has attracted increasing attention in recent years. The main challenge is to overcome the distribution shift of “action-objects” pairs between the training and testing sets. Previous works for CAR usually introduce extra information (e.g. bounding box) to enhance the dynamic cues of video features. However, these approaches do not essentially eliminate the inherent inductive bias in the video, which can be regarded as the stumbling block for model generalization. Because the video features are usually extracted from the visually cluttered areas in which many objects cannot be removed or masked explicitly. To this end, this work attempts to implicitly accomplish semantic-level decoupling of “object-action” in the high-level feature space. Specifically, we propose a novel Semantic-Decoupling Transformer framework, dubbed as DeFormer, which contains two insightful sub-modules: Objects-Motion Decoupler (OMD) and Semantic-Decoupling Constrainer (SDC). In OMD, we initialize several learnable tokens incorporating annotation priors to learn an instance-level representation and then decouple it into the appearance feature and motion feature in high-level visual space. In SDC, we use textual information in the high-level language space to construct a dual-contrastive association to constrain the decoupled appearance feature and motion feature obtained in OMD. Extensive experiments verify the generalization ability of DeFormer. Specifically, compared to the baseline method, DeFormer achieves absolute improvements of 3%, 3.3%, and 5.4% under three different settings on STH-ELSE, while corresponding improvements on EPIC-KITCHENS-55 are 4.7%, 9.2%, and 4.4%. Besides, DeFormer gains state-of-the-art results either on ground-truth or detected annotations.Keywords
Funding Information
- National Key Research and Development Program of China (2018AAA0102002)
- National Natural Science Foundation of China (61925204, 62222207)
- Natural Science Foundation of Jiangsu Province (BK20211520)
This publication has 56 references indexed in Scilit:
- Skeleton-Based Human Action Recognition With Global Context-Aware Attention LSTM NetworksIEEE Transactions on Image Processing, 2017
- Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust GatesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2017
- Global Context-Aware Attention LSTM Networks for 3D Action Recognition2017
- From Red Wine to Red Tomato: Composition with Context2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding2016
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition2016
- Convolutional Two-Stream Network Fusion for Video Action Recognition2016
- Learning Spatiotemporal Features with 3D Convolutional Networks2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015