External Validation and Calibration Assessment of Explainable Machine Learning Models for GVHD Prediction After Allogeneic HSCT
Authors:
Preprint
Abstract
Background: Graft-versus-host disease (GVHD) remains a major determinant of morbidity and mortality following allogeneic hematopoietic stem cell transplantation (allo-HSCT). Existing GVHD prediction models demonstrate modest discrimination and limited generalizability, and calibration drift across external populations is rarely characterized despite its essential role in the clinical interpretability of predicted probabilities. Objectives: To develop and externally validate an explainable machine learning framework for predicting acute and chronic GVHD and associated overall survival in patients with acute myeloid leukemia (AML), acute lymphoblastic leukemia (ALL), and myelodysplastic syndromes (MDS) undergoing allo-HSCT, and to systematically characterize calibration across heterogeneous external validation cohorts to inform deployment requirements. Study Design: The model was developed on three publicly available registry-derived datasets (N = 2,509) and externally validated across six independent cohorts (N = 14,788) comprising adult and pediatric allo-HSCT recipients, including a regional Middle Eastern cohort (UAE and Jordan). A standardized preprocessing pipeline harmonized heterogeneous datasets. Gradient boosting models (CatBoost) were used for binary GVHD prediction; exploratory overall survival analysis used a Cox proportional hazards model with predicted acute GVHD risk as a covariate. Discrimination (AUROC with bootstrap 95% CI), calibration (logistic recalibration intercept and slope with analytical 95% CI), and feature importance (SHapley Additive exPlanations, SHAP) were assessed in training out-of-fold and all external cohorts. Results: In internal validation, AUROC was 0.63 (95% CI 0.61–0.65) for acute GVHD and 0.72 (95% CI 0.70–0.74) for chronic GVHD. External validation demonstrated AUROC ranges of 0.51–0.57 (acute) and 0.54–0.64 (chronic), with consistent performance across disease subgroups despite substantial heterogeneity in transplant practices and feature availability. In exploratory survival analysis, the acute-GVHD-informed Cox model achieved a training-cohort C-index of 0.679 (95% CI 0.658–0.697); external C-indices ranged from 0.47–0.53. Calibration analysis identified systematic external risk overestimation (negative calibration intercept in 10 of 11 evaluable external cohort-target combinations) with heterogeneous slope drift requiring cohort-specific recalibration. Key predictors included recipient age, graft source, conditioning intensity, GVHD prophylaxis, and HLA match ratio. Conclusions: An explainable, externally validated GVHD prediction framework was developed using heterogeneous registry-derived datasets, with systematic characterization of calibration drift across multiple external cohorts, an analysis rarely reported in prior GVHD prediction literature. Predictive performance was modest for acute GVHD and moderate for chronic GVHD, constrained by missing immunobiological variables and incomplete HLA characterization. Per-cohort recalibration is required before clinical deployment, with prospective validation and benchmarking against established GVHD risk scores identified as priority next steps.Keywords
This publication has 30 references indexed in Scilit:
- Harnessing Machine Learning to Predict Chronic Gvhd: A Novel Risk Stratification ScoreTransplantation and Cellular Therapy, 2025
- A novel prognostic scoring system for AML patients undergoing allogeneic hematopoietic stem cell transplantation with real world validationJournal of Advanced Research, 2024
- Return to Work Among Young Adult Survivors of Allogeneic Hematopoietic Cell Transplantation in the United StatesTransplantation and Cellular Therapy, 2021
- Specific Class I HLA Supertypes but Not HLA Zygosity or Expression Are Associated with Outcomes following HLA-Matched Allogeneic Hematopoietic Cell Transplant: HLA Supertypes Impact Allogeneic HCT OutcomesTransplantation and Cellular Therapy, 2021
- Optimal Donor for African Americans with Hematologic Malignancy: HLA-Haploidentical Relative or Umbilical Cord Blood TransplantTransplantation and Cellular Therapy, 2020
- Risk Factors for Graft-versus-Host Disease in Haploidentical Hematopoietic Cell Transplantation Using Post-Transplant CyclophosphamideTransplantation and Cellular Therapy, 2020
- Acute Graft-versus-Host Disease — Biologic Process, Prevention, and TherapyNew England Journal of Medicine, 2017
- An early-biomarker algorithm predicts lethal graft-versus-host disease and survivalJCI Insight, 2017
- Graft-versus-host diseaseThe Lancet, 2009
- Chronic graft-versus-host diseaseTransplantation and Cellular Therapy, 2003