MMW-AQA: Multimodal In-the-Wild Dataset for Action Quality Assessment
Open Access
- 4 July 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 92062-92072
Abstract
Action quality assessment (AQA) is a task for assessing a specific action quality in videos. Since existing AQA datasets provide only two-dimensional (2D) video data captured from fewer viewpoints, existing AQA methods based on deep neural networks (DNNs) struggle to assess complex three-dimensional (3D) actions accurately, and their robustness against diversified viewpoints remains unknown. We created a dataset called multimodal in-the-wild (MMW)-AQA in freestyle windsurfing that addresses these concerns. In addition to video data, MMW-AQA provides inertial measurement unit (IMU) and global positioning system (GPS) data. The 3D information of IMU data helps DNNs accurately assess complex 3D actions. Moreover, MMW-AQA provides wild video data captured by an unmanned aerial vehicle (UAV). These wild video data enable us to evaluate whether AQA methods can work well on diversified viewpoints. Furthermore, we also present the baseline multimodalization framework with a transformer-based fusion module. These frameworks multimodalize existing unimodal DNN models easily to assess action quality using multimodal data. Our experimental results demonstrate that multimodal data improves the AQA accuracy compared with unimodal video data.Keywords
This publication has 37 references indexed in Scilit:
- Automatic Modelling for Interactive Action AssessmentInternational Journal of Computer Vision, 2022
- Action Quality Assessment with Temporal Parsing Transformer2022
- Pairwise Contrastive Learning Network for Action Quality Assessment2022
- Gaussian guided frame sequence encoder network for action quality assessmentComplex & Intelligent Systems, 2022
- The Pros and Cons: Rank-Aware Temporal Attention for Skill Determination in Long Videos2019
- What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessment2019
- Who's Better? Who's Best? Pairwise Deep Ranking for Skill Determination2018
- Temporal Convolutional Networks for Action Segmentation and Detection2017
- Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset2017
- Automated video-based assessment of surgical skills for training and evaluation in medical schoolsInternational Journal of Computer Assisted Radiology and Surgery, 2016