SkatingVerse: A large‐scale benchmark for comprehensive evaluation on human action understanding
Authors:
Open Access
- 30 May 2024
- journal article
- research article
- Published by Institution of Engineering and Technology (IET) in IET Computer Vision
- Vol. 18 (7) , 888-906
Abstract
Human action understanding (HAU) is a broad topic that involves specific tasks, such as action localisation, recognition, and assessment. However, most popular HAU datasets are bound to one task based on particular actions. Combining different but relevant HAU tasks to establish a unified action understanding system is challenging due to the disparate actions across datasets. A large‐scale and comprehensive benchmark, namely SkatingVerse is constructed for action recognition, segmentation, proposal, and assessment. SkatingVerse focus on fine‐grained sport action, hence figure skating is chosen as the task object, which eliminates the biases of the object, scene, and space that exist in most previous datasets. In addition, skating actions have inherent complexity and similarity, which is an enormous challenge for current algorithms. A total of 1687 official figure skating competition videos was collected with a total of 184.4 h, exceeding four times over other datasets with a similar topic. SkatingVerse enables to formulate a unified task to output fine‐grained human action classification and assessment results from a raw figure skating competition video. In addition, SkatingVerse can facilitate the study of HAU foundation model due to its large scale and abundant categories. Moreover, image modality is incorporated for human pose estimation task into SkatingVerse . Extensive experimental results show that (1) SkatingVerse significantly helps the training and evaluation of HAU methods, (2) the performance of existing HAU methods has much room to improve, and SkatingVerse helps to reduce such gaps, and (3) unifying relevant tasks in HAU through a uniform dataset can facilitate more practical applications. SkatingVerse will be publicly available to facilitate further studies on relevant problems.Keywords
Funding Information
- National Natural Science Foundation of China (62006244, 62102039)
This publication has 40 references indexed in Scilit:
- AdaptivePose: Human Parts as Adaptive PointsProceedings of the AAAI Conference on Artificial Intelligence, 2022
- Video Swin Transformer2022
- X3D: Expanding Architectures for Efficient Video Recognition2020
- What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessment2019
- BSN: Boundary Sensitive Network for Temporal Action Proposal Generation2018
- Temporal Convolutional Networks for Action Segmentation and Detection2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition2016
- Deep Residual Learning for Image Recognition2016
- The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities2014