Early-Horizon Multimodal ICU Mortality Prediction Without Retraining
Preprint
Abstract
Earlier ICU mortality prediction is more clinically useful because it can identify high-risk patients while treatment decisions can still change. Yet most models are trained on data from a fixed time window, so it is unclear whether a model trained on the first 48 hours of ICU data remains reliable when used earlier in the ICU stay. We evaluated a multimodal ICU mortality model trained once at 48 hours and then applied unchanged at 6, 12, 24, and 48 hours on MIMIC-III. The model combines an LSTM for physiological time-series data, a finetuned ClinicalModernBERT model for clinical notes, and a logistic regression fusion layer. Performance remained strong at earlier time points, suggesting that useful mortality prediction is possible earlier in the ICU stay even without retraining. At 6 hours, the model achieved AUROC 0.777 and remained well-calibrated (expected calibration error, ECE 0.038) without any recalibration, and it outperformed both single-modality models at every horizon. The benefit of combining both modalities was most evident at earlier horizons, when physiological data were sparse: agreement between the two specialists dropped by more than half from 48 to 6 hours, while the median contribution from clinical notes increased from 37% to 49%. A Bayesian version of the fusion layer showed that uncertainty decreased for survivors as more data accumulated but remained high for non-survivors; the most uncertain cases were up to 4.9 times more likely to be non-surviving patients. Continuous hourly analyses further showed that clinical notes provide stable context between documentation events. Simply carrying forward the most recent note matched or outperformed note-decay and documentation-gap alternatives. These results suggest that a multimodal ICU mortality model trained on 48 hours of data can provide trustworthy earlier predictions without retraining, while also identifying the cases that remain hardest to interpret.Keywords
This publication has 17 references indexed in Scilit:
- A manifesto on explainability for artificial intelligence in medicineArtificial Intelligence in Medicine, 2022
- From development to deployment: dataset shift, causality, and shift-stable models in health AIBiostatistics, 2019
- Multitask learning and benchmarking with clinical time series dataScientific Data, 2019
- High-performance medicine: the convergence of human and artificial intelligenceNature Medicine, 2019
- The need for uncertainty quantification in machine-assisted medical decision makingNature Machine Intelligence, 2019
- MIMIC-III, a freely accessible critical care databaseScientific Data, 2016
- Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble AccuracyMachine Learning, 2003
- Bidirectional recurrent neural networksIEEE Transactions on Signal Processing, 1997
- Stacked RegressionsMachine Learning, 1996
- Stacked generalizationNeural Networks, 1992