Language Models Are An Effective Patient Representation Learning Technique For Electronic Health Record Data
Authors:
- Ethan SteinbergESEthan Steinberg
- Ken JungKJKen Jung
- Jason A. FriesJFJason A. Fries
- Conor K. CorbinCCConor K. Corbin
- Stephen R. PfohlSPStephen R. Pfohl
- Nigam H. ShahNSNigam H. Shah
Preprint
Abstract
Widespread adoption of electronic health records (EHRs) has fueled development of clinical outcome models using machine learning. However, patient EHR data are complex, and how to optimally represent them is an open question. This complexity, along with often small training set sizes available to train these clinical outcome models, are two core challenges for training high quality models. In this paper, we demonstrate that learning generic representations from the data of all the patients in the EHR enables better performing prediction models for clinical outcomes, allowing for these challenges to be overcome. We adapt common representation learning techniques used in other domains and find that representations inspired by language models enable a 3.5% mean improvement in AUROC on five clinical outcomes compared to standard baselines, with the average improvement rising to 19% when only a small number of patients are available for training a prediction model for a given clinical outcome.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Deep learning and alternative learning strategies for retrospective real-world clinical datanpj Digital Medicine, 2019
- Finding missed cases of familial hypercholesterolemia in health systems using machine learningnpj Digital Medicine, 2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Patient2Vec: A Personalized Interpretable Deep Representation of the Longitudinal Electronic Health RecordIEEE Access, 2018
- Implementation of the Sepsis Risk Calculator at an Academic Birth HospitalHospital Pediatrics, 2018
- Using recurrent neural network models for early detection of heart failure onsetJournal of the American Medical Informatics Association, 2016
- Deep Patient: An Unsupervised Representation to Predict the Future of Patients from the Electronic Health RecordsScientific Reports, 2016
- The Unified Medical Language System (UMLS): integrating biomedical terminologyNucleic Acids Research, 2004
- Using Linear Algebra for Intelligent Information RetrievalSIAM Review, 1995
- Singular value decomposition and least squares solutionsNumerische Mathematik, 1970