EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models
Preprint
Abstract
While the general machine learning (ML) community has benefited from public datasets, tasks, and models, the progress of ML in healthcare has been hampered by a lack of such shared assets. The success of foundation models creates new challenges for healthcare ML by requiring access to shared pretrained models to validate performance benefits. We help address these challenges through three contributions. First, we publish a new dataset, EHRSHOT, which contains deidentified structured data from the electronic health records (EHRs) of 6,739 patients from Stanford Medicine. Unlike MIMIC-III/IV and other popular EHR datasets, EHRSHOT is longitudinal and not restricted to ICU/ED patients. Second, we publish the weights of CLMBR-T-base, a 141M parameter clinical foundation model pretrained on the structured EHR data of 2.57M patients. We are one of the first to fully release such a model for coded EHR data; in contrast, most prior models released for clinical data (e.g. GatorTron, ClinicalBERT) only work with unstructured text and cannot process the rich, structured data within an EHR. We provide an end-to-end pipeline for the community to validate and build upon its performance. Third, we define 15 few-shot clinical prediction tasks, enabling evaluation of foundation models on benefits such as sample efficiency and task adaptation. Our model and dataset are available via a research data use agreement from our website: https://ehrshot.stanford.edu. Code to reproduce our results are available at our Github repo: https://github.com/som-shahlab/ehrshot-benchmarkKeywords
All Related Versions
This publication has 16 references indexed in Scilit:
- RoFormer: Enhanced transformer with Rotary Position EmbeddingNeurocomputing, 2023
- EHR foundation models improve robustness in the presence of temporal distribution shiftScientific Reports, 2023
- The reproducibility issues that haunt health-care AINature, 2023
- Addressing bias in big data and AI for health care: A call for open sciencePatterns, 2021
- Deep learning for chest X-ray analysis: A surveyMedical Image Analysis, 2021
- Language models are an effective representation learning technique for electronic health record dataJournal of Biomedical Informatics, 2020
- Potential Biases in Machine Learning Algorithms Using Electronic Health Record DataJAMA Internal Medicine, 2018
- The UK Biobank resource with deep phenotyping and genomic dataNature, 2018
- Benchmarking deep learning models on large healthcare datasetsJournal of Biomedical Informatics, 2018
- Scalable and accurate deep learning with electronic health recordsnpj Digital Medicine, 2018