Quantifying the imputation paradox and XAI inconsistency in multi-source diabetes prediction: a 353,680-record leakage-free stacking ensemble with dynamic routing architecture
Authors:
Open Access
- 7 August 2026
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Artificial Intelligence
- Vol. 9, 1898153
Abstract
Diabetes affects 537 million adults globally, a figure projected to reach 783 million by 2045. Despite over 4,200 ML prediction studies, clinical translation remains hindered by an over-reliance on benchmark datasets, unmeasured information costs of multi-source fusion, and untested XAI convergence assumptions. We address these issues by evaluating 353,680 records (from 455,446 candidates) across a Clinical-Biomarker Set (CBS) and a Lifestyle-Survey Set (LSS) using a strict leakage-free protocol. Under this protocol, normalisation, imputation, and SMOTE synthesis were fitted exclusively on training data. We formally define the Imputation Paradox as the AUC loss that occurs when critical diagnostic biomarkers are median-imputed under structural MNAR missingness. An Optuna-tuned stacking ensemble (Random Forest, XGBoost, LightGBM) achieves an AUC-ROC of 0.8481 on a held-out 70,736-sample test set. While a simple average baseline marginally exceeded stacking on pure discrimination (AUC 0.8521), stacking was retained for its superior precision (0.3439) and interpretable meta-learner trust weights, yielding a cross-validation AUC of 0.8491 ± 0.0014 and a Wilcoxon-significant improvement over baselines ( p < 0.001, n = 1,000 replicates). A CBS sub-model trained on complete HbA1c and glucose data achieves an AUC of 0.9781. The resulting 0.130-unit gap quantifies this fusion-induced information cost. To recover this, our end-to-end dynamic routing pipeline routes 28.1% of patients to the CBS sub-model and 71.9% to the stacking ensemble, achieving a blended AUC of 0.8496. At a 0.30 mass screening threshold, the model identifies ~16,600 additional diabetic patients per million screened, with decision curve analysis confirming net benefit across the 5–30% threshold range. XAI conflict analysis yields Spearman correlations of r = −0.4561 (SHAP/permutation, p = 0.141), r = +0.2417 (SHAP/LIME, p = 0.737), and r = +0.1494 (permutation/LIME, p = 0.350). While non-significant at n = 12 features (critical r = ±0.576, α = 0.05), this provides preliminary directional evidence that these methods capture structurally different aspects of feature relevance.Keywords
This publication has 33 references indexed in Scilit:
- Leakage and the reproducibility crisis in machine-learning-based sciencePatterns, 2023
- Likelihood Prediction of Diabetes at Early Stage Using Data Mining Techniques2019
- Causability and explainability of artificial intelligence in medicineWIREs Data Mining and Knowledge Discovery, 2019
- A systematic study of the class imbalance problem in convolutional neural networksNeural Networks, 2018
- Comparative approaches for classification of diabetes mellitus data: Machine learning paradigmComputer Methods and Programs in Biomedicine, 2017
- Tree-based classifier ensembles for early detection method of diabetes: an exploratory studyArtificial Intelligence Review, 2017
- Learning from class-imbalanced data: Review of methods and applicationsExpert Systems with Applications, 2017
- Machine Learning and Data Mining Methods in Diabetes ResearchComputational and Structural Biotechnology Journal, 2017
- Missing data imputation using statistical and machine learning methods in a real breast cancer problemArtificial Intelligence in Medicine, 2010
- A theory of learning from different domainsMachine Learning, 2009