Exploring the Potential and Limitations of Deep Learning and Explainable AI for Longitudinal Life Course Analysis
Preprint
- 27 March 2025
- preprint
- Published by Springer Nature in Research Square
Abstract
Background: Understanding the complex interplay between life course exposures, such as adverse childhood experiences and environmental factors, and disease risk is essential for developing effective public health interventions. Traditional epidemiological methods, such as regression models and risk scoring, are limited in their ability to capture the non-linear and temporally dynamic nature of these relationships. Deep learning (DL) and explainable artificial intelligence (XAI) are increasingly applied within healthcare settings to identify influential risk factors and enable personalised interventions. However, significant gaps remain in understanding their utility and limitations, especially for sparse longitudinal life course data and how the influential patterns identified using explainability are linked to underlying causal mechanisms. Methods: We conducted a controlled simulation study to assess the performance of various state-of-the-art DL architectures including CNNs and (attention-based) RNNs against XGBoost and logistic regression. Input data was simulated to reflect a generic and generalisable scenario with different rules used to generate multiple realistic outcomes based upon epidemiological concepts. Multiple metrics were used to assess model performance in the presence of class imbalance and SHAP values were calculated. Results: We find that DL methods can accurately detect dynamic relationships that baseline linear models and tree-based methods cannot. However, there is no one model that consistently outperforms the others across all scenarios. We further identify the superior performance of DL models in handling sparse feature availability over time compared to traditional machine learning approaches. Additionally, we examine the interpretability provided by SHAP values, demonstrating that these explanations often misalign with causal relationships, despite excellent predictive and calibrative performance. Conclusions: These insights provide a foundation for future research applying DL and XAI to life course data, highlighting the challenges associated with sparse healthcare data, and the critical need for advancing interpretability frameworks in personalised public health.Keywords
All Related Versions
- Version 1, 2025-03-27, Research Square
- Published version: BMC Public Health, 25 (1), 1.
This publication has 35 references indexed in Scilit:
- Comparing AI/ML approaches and classical regression for predictive modeling using large population health databases: Applications to COVID-19 case predictionGlobal Epidemiology, 2024
- From patterns to patients: Advances in clinical machine learning for cancer diagnosis, prognosis, and treatmentCell, 2023
- Tabular data: Deep learning is not all you needInformation Fusion, 2022
- Explainable deep learning in healthcare: A methodological survey from an attribution viewWIREs Mechanisms of Disease, 2022
- Statistical Modeling of Sensitive Period Effects Using the Structured Life Course Modeling Approach (SLCMA)2021
- Explainable machine learning prediction of ICU mortalityInformatics in Medicine Unlocked, 2021
- The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advancesData Mining and Knowledge Discovery, 2020
- Taking the aggravation out of data aggregation: A conceptual guide to dealing with statistical issues related to the pooling of individual‐level observational dataAmerican Journal of Primatology, 2015
- The Cumulative Effect of Unemployment on Risks for Acute Myocardial InfarctionArchives of internal medicine (1960), 2012
- The Emergence and Development of Life Course Theory2003