Rhythmer: Ranking-Based Skill Assessment With Rhythm-Aware Transformer
- 13 September 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (1) , 259-272
Abstract
Ranking-based skill assessment is an essential component of video understanding. In this task lacking precise procedure annotations, existing methods place greater emphasis on evaluating the procedure quality via manually normalizing the execution duration. However, the inherent duration-related procedural patterns will undergo alteration. Experimentally, we discover that distinct duration biases are prevalent in duration-sensitive skills, such as those in medical and everyday life. Hence, duration information is crucial for ranking-based skill assessment when dealing with varying durations. Additionally, similar execution processes tend to have closer execution durations. Thus, another critical factor lies in extracting duration-related procedural information alongside similar durations. It is defined as mining rhythm patterns, which are inspired by music rhythms including various duration and duration-related procedures. In our work, a rhythm-aware transformer is proposed to mine the rhythm patterns adaptively. Given pairwise inputs, a coattention module is designed to mutually highlight duration-related procedure information when comparing pairwise input videos with similar durations, and adaptively attenuate the efficacy when confronted with pairwise inputs featuring significantly different durations. A rhythm-encoding module further embeds duration information into the concatenation of raw features and co-attention features. Following these features, the transformer decoder is designed to learn duration-related queries supervised by a novel duration grouping loss among various duration groups. The experimental results demonstrate that the rhythm-aware transformer is effective for ranking-based skill assessment.Keywords
Funding Information
- National Natural Science Foundation of China (62371219, 62271221, 61771233)
- Basic and Applied Basic Research Foundation of Guangdong Province (2023A1515011260)
- Science and Technology Program of Guangzhou (202201011672)
- Science and Engineering Research Council (SERC) Central Research Fund
This publication has 48 references indexed in Scilit:
- Action Quality Assessment with Temporal Parsing Transformer2022
- Pairwise Contrastive Learning Network for Action Quality Assessment2022
- Analysis of executional and procedural errors in dry‐lab robotic surgery experimentsThe International Journal of Medical Robotics and Computer Assisted Surgery, 2022
- Clearness of operating field: a surrogate for surgical skills on in vivo clinical dataInternational Journal of Computer Assisted Radiology and Surgery, 2020
- Surgical Skill Assessment on In-Vivo Clinical Data via the Clearness of Operating Field2019
- Laparoscopic training using a quantitative assessment and instructional systemInternational Journal of Computer Assisted Radiology and Surgery, 2018
- Automated video-based assessment of surgical skills for training and evaluation in medical schoolsInternational Journal of Computer Assisted Radiology and Surgery, 2016
- Procedural surgical skill assessment in laparoscopic training environmentsInternational Journal of Computer Assisted Radiology and Surgery, 2015
- Assessing the Quality of Actions2014
- Pairwise Comparison-Based Objective Score for Automated Skill Assessment of Segments in a Surgical Task2014