Subject-Aware Model Validation for Repeated-Measures Data: A Nested Approach for Trustworthy Medical AI Applications (Preprint)
Preprint
- 13 November 2025
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND Repeated-measures datasets are common in biomechanics and digital health, where each participant contributes multiple correlated trials. If cross-validation (CV) ignores this structure, information can leak from training to test folds, inflating performance and undermining clinical credibility. OBJECTIVE To evaluate the impact of subject-aware validation strategies on model reliability in repeated-measures classification tasks, using fear of re-injury prediction post–anterior cruciate ligament reconstruction (ACLR) as a case study. METHODS We analyzed 623 hop trials from 72 individuals post-ACLR to classify fear of re-injury based on biomechanical features. Four cross-validation (CV) strategies were compared: stratified 10-fold CV, Leave-One-Participant-Out CV (LOPOCV), Group 3-Fold CV, and a nested framework combining LOPOCV (outer loop) with Group 3-fold CV (inner loop). Ten supervised classifiers were benchmarked across classification accuracy, train–test generalization gap, model ranking consistency, and computational efficiency. RESULTS Stratified 10-Fold CV systematically overestimated model performance (e.g., Extra Trees accuracy of 0.91 vs. 0.66 under LOPOCV) due to subject-level data leakage. Group and nested CV strategies yielded more conservative and stable estimates. The nested LOPOCV + Group CV framework achieved a good balance between generalization and participant-level independence, with reduced bias and overfitting compared to non-nested alternatives. CONCLUSIONS Subject-aware validation strategies are essential for trustworthy ML evaluation in repeated-measures settings. Nested CV designs improve reproducibility, reduce selection bias, and align with regulatory expectations for clinical ML tools. These findings support best practices in model validation for biomechanics and digital health applications.Keywords
This publication has 50 references indexed in Scilit:
- Navigating prevalence shifts in image analysis algorithm deploymentMedical Image Analysis, 2025
- Characteristics of Rank‐based Penalty Estimators2022
- Experimental evaluation of genomic selection prediction for rust resistance in sugarcaneThe Plant Genome, 2021
- Establishment of Best Practices for Evidence for PredictionJAMA Psychiatry, 2020
- Machine learning in human movement biomechanics: Best practices, common pitfalls, and new opportunitiesJournal of Biomechanics, 2018
- On Splitting Training and Validation Set: A Comparative Study of Cross-Validation, Bootstrap and Systematic Sampling for Estimating the Generalization Performance of Supervised LearningJournal of Analysis and Testing, 2018
- Prediction models need appropriate internal, internal–external, and external validationJournal of Clinical Epidemiology, 2016
- Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of powerInformation Sciences, 2010
- Random Effects Models for Longitudinal Data2010
- The reliability of three-dimensional kinematic gait measurements: A systematic reviewGait & Posture, 2009