CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks
Preprint
Abstract
Embedding algorithms are increasingly used to represent clinical concepts in healthcare for improving machine learning tasks such as clinical phenotyping and disease prediction. Recent studies have adapted state-of-the-art bidirectional encoder representations from transformers (BERT) architecture to structured electronic health records (EHR) data for the generation of contextualized concept embeddings, yet do not fully incorporate temporal data across multiple clinical domains. Therefore we developed a new BERT adaptation, CEHR-BERT, to incorporate temporal information using a hybrid approach by augmenting the input to BERT using artificial time tokens, incorporating time, age, and concept embeddings, and introducing a new second learning objective for visit type. CEHR-BERT was trained on a subset of Columbia University Irving Medical Center-York Presbyterian Hospital's clinical data, which includes 2.4M patients, spanning over three decades, and tested using 4-fold cross-validation on the following prediction tasks: hospitalization, death, new heart failure (HF) diagnosis, and HF readmission. Our experiments show that CEHR-BERT outperformed existing state-of-the-art clinical BERT adaptations and baseline models across all 4 prediction tasks in both ROC-AUC and PR-AUC. CEHR-BERT also demonstrated strong transfer learning capability, as our model trained on only 5% of data outperformed comparison models trained on the entire data set. Ablation studies to better understand the contribution of each time component showed incremental gains with every element, suggesting that CEHR-BERT's incorporation of artificial time tokens, time and age embeddings with concept embeddings, and the addition of the second learning objective represents a promising approach for future BERT-based clinical embeddings.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease predictionnpj Digital Medicine, 2021
- A comprehensive EHR timeseries pre-training benchmark2021
- BEHRT: Transformer for Electronic Health RecordsScientific Reports, 2020
- Temporal Self-Attention Network for Medical Concept Embedding2019
- Time-sensitive clinical concept embeddings learned from large electronic health recordsBMC Medical Informatics and Decision Making, 2019
- Patient2Vec: A Personalized Interpretable Deep Representation of the Longitudinal Electronic Health RecordIEEE Access, 2018
- A machine learning model to predict the risk of 30-day readmissions in patients with heart failure: a retrospective analysis of electronic medical records dataBMC Medical Informatics and Decision Making, 2018
- Recurrent Neural Networks for Multivariate Time Series with Missing ValuesScientific Reports, 2018
- Early Detection of Heart Failure Using Electronic Health RecordsCirculation: Cardiovascular Quality and Outcomes, 2016
- Observational Health Data Sciences and Informatics (OHDSI): Opportunities for Observational Researchers2015