Causality-Enhanced Multiple Instance Learning With Graph Convolutional Networks for Parkinsonian Freezing-of-Gait Assessment
- 24 June 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 3991-4001
Abstract
Freezing of gait (FoG) is a common disabling symptom of Parkinson’s disease (PD). It is clinically characterized by sudden and transient walking interruptions for specific human body parts, and it presents the localization in time and space. Due to the difficulty in extracting global fine-grained features from lengthy videos, developing an automated five-point FoG scoring system is quite challenging. Therefore, we propose a novel video-based automated five-classification FoG assessment method with a causality-enhanced multiple-instance-learning graph convolutional network (GCN). This method involves developing a temporal segmentation GCN to segment each video into three motion stages for stage-level feature modeling, followed by a multiple-instance-learning framework to divide each stage into short clips for instance-level feature extraction. Subsequently, an uncertainty-driven multiple-instance-learning GCN is developed to capture spatial and temporal fine-grained features through GCN scheme and uncertainty learning, respectively, for acquiring global representations. Finally, a causality-enhanced graph generation strategy is proposed to exploit causal inference for mining and enhancing human structures causally related to clinical assessment, thereby extracting spatial causal features. Extensive experimental results demonstrate the excellent performance of the proposed method on five-classification FoG assessment with an accuracy of 62.72% and an acceptable accuracy of 91.32%, which is confirmed by independent testing. Additionally, it enables temporal and spatial localization of FoG events to a certain extent, facilitating reasonable clinical interpretations. In conclusion, our method provides a valuable tool for automated FoG assessment in PD, and the proposed causality-related component exhibits promising potential for extension to other general and medical fine-grained action recognition tasks.Keywords
Funding Information
- Guangxi Key Research and Development Project (2023AB22106)
- Special Fund for Transformation of Scientific and Technological Achievements of Inner Mongolia Autonomous Region of China (2021CG0052)
- National Natural Science Foundation of China (62371286, 62171273)
- Natural Science Foundation of Shanghai Municipality (22ZR1432100)
This publication has 49 references indexed in Scilit:
- A tree-structure-guided graph convolutional network with contrastive learning for the assessment of parkinsonian hand movementsMedical Image Analysis, 2022
- Automated assessment of Parkinsonian finger-tapping tests through a vision-based fine-grained classification modelNeurocomputing, 2021
- FSD-10: A fine-grained classification dataset for figure skatingNeurocomputing, 2020
- Detecting freezing of gait with a tri-axial accelerometer in Parkinson’s disease patientsMedical & Biological Engineering & Computing, 2015
- Clinical assessment of freezing of gait in Parkinson's disease from computer-generated animationGait & Posture, 2013
- Movement Disorder Society‐sponsored revision of the Unified Parkinson's Disease Rating Scale (MDS‐UPDRS): Scale presentation and clinimetric testing resultsMovement Disorders, 2008
- Freezing of gait affects quality of life of peoples with Parkinson's disease beyond its relationships with mobility and gaitMovement Disorders, 2007
- Predictors of freezing in Parkinson's disease: A survey of 6,620 patientsMovement Disorders, 2007
- Freezing of gait in Parkinson’s diseaseJournal of Neurology, 2006
- Unified Parkinson's disease rating scale motor examination: Are ratings of nurses, residents in neurology, and movement disorders specialists interchangeable?Movement Disorders, 2005