FERMAT: An Autoregressive Clinical Foundation Model for Disease Risk and Clinical Event Prediction
Preprint
Abstract
Clinical foundation models learn patterns in longitudinal health records that can be reused across prediction tasks. However, the relative strengths of representation-based and generation-based prediction across clinical tasks remain unclear. Using longitudinal electronic health records from 3.77 million patients at a single hospital, we developed FERMAT (Foundation model for Exploring Real-world Multimodal health data using Autoregressive Trajectory modeling), a 65-million-parameter clinical foundation model. We compared predictions from task-specific predictors trained on patient representations with those derived from generated clinical event sequences. Representation-based prediction performed better in the disease-risk comparisons. In detailed analyses of five-year chronic kidney disease, fatty liver, and diabetes risk in a test cohort of 10,000 patients, representation-based predictors exceeded generation-based estimates by 0.098–0.115 in observation-weighted AUROC and had lower probability error. Generation-based prediction, in turn, performed better in predicting diagnoses, prescriptions, and procedures recorded on the same day, a task motivated by the frequent presence of multiple events with the same date in hospital EHRs. Using 50 generated trajectories per patient, it achieved a Recall@10 of 71.0%, compared with 55.7% for representation-based prediction, with a paired improvement of 15.31 percentage points (95% CI, 13.92–16.79; n = 1,862). These findings demonstrate task-dependent strengths of representation-based and generation-based applications within a single clinical foundation model, providing an empirical basis for exploring how the two approaches can be used individually or combined to address clinical prediction needs.Keywords
Funding Information
- Korea Health Industry Development Institute (RS-2025-25462804)
This publication has 7 references indexed in Scilit:
- Learning the natural history of human disease with generative transformersNature, 2025
- Zero shot health trajectory prediction using transformernpj Digital Medicine, 2024
- Foresight—a generative pretrained transformer for modelling of patient timelines using electronic health records: a retrospective modelling studyThe Lancet Digital Health, 2024
- Foundation models for generalist medical artificial intelligenceNature, 2023
- EHR foundation models improve robustness in the presence of temporal distribution shiftScientific Reports, 2023
- Language models are an effective representation learning technique for electronic health record dataJournal of Biomedical Informatics, 2020
- Regression Models and Life-TablesJournal of the Royal Statistical Society Series B: Statistical Methodology, 1972