Video-Language Graph Convolutional Network for Human Action Recognition
Authors:
- 14 April 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7995-7999
Abstract
Transferring visual language models (VLMs) from the image domain to the video domain has recently yielded great success on human action recognition tasks. However, standard recognition paradigms overlook fine-grained action parsing knowledge that could enhance the recognition accuracy. In this paper, we propose a novel method that leverages both coarse-grained and fine-grained knowledge to recognize human actions in videos. Our method consists of a video-language graph convolutional network that integrates and fuses multi-modal knowledge in a progressive manner. We evaluate our method on the Kinetics-TPS, a large-scale action parsing dataset, and demonstrate that it outperforms the state-of-the-art methods by a significant margin. Moreover, our method achieves better results with less training data and competitive computational cost than the existing methods, showing the effectiveness and efficiency of using fine-grained knowledge for human video action recognition.Keywords
This publication has 13 references indexed in Scilit:
- Prompting Visual-Language Models for Efficient Video Understanding2022
- Expanding Language-Image Pretrained Models for General Video Recognition2022
- A Simple Meta-path-free Framework for Heterogeneous Network Embedding2022
- Visual Knowledge Graph for Human Action Reasoning in Videos2022
- Multiview Transformers for Video Recognition2022
- Video Swin Transformer2022
- Self-Regulated Learning for Egocentric Video Activity AnticipationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- A survey on video-based Human Action Recognition: recent updates, datasets, challenges, and applicationsArtificial Intelligence Review, 2020
- PaStaNet: Toward Human Activity Knowledge Engine2020
- Temporal Segment Networks for Action Recognition in VideosIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018