Moving Beyond Medical Statistics: A Systematic Review on Missing Data Handling in Electronic Health Records
Authors:
- Shenda HongSHShenda Hong
- Huixin LiuHLHuixin Liu
- on behalf of the Missing Data in Electronic health Records (MINDER) GroupOGon behalf of the Missing Data in Electronic health Records (MINDER) Group
Top Cited Papers
Open Access
- 1 January 2024
- journal article
- review article
- Published by American Association for the Advancement of Science (AAAS) in Health Data Science
- Vol. 4, 0176
Abstract
Background: Missing data in electronic health records (EHRs) presents significant challenges in medical studies. Many methods have been proposed, but uncertainty exists regarding the current state of missing data addressing methods applied for EHR and which strategy performs better within specific contexts. Methods: All studies referencing EHR and missing data methods published from their inception until 2024 March 30 were searched via the MEDLINE, EMBASE, and Digital Bibliography and Library Project databases. The characteristics of the included studies were extracted. We also compared the performance of various methods under different missingness scenarios. Results: After screening, 46 studies published between 2010 and 2024 were included. Three missingness mechanisms were simulated when evaluating the missing data methods: missing completely at random (29/46), missing at random (20/46), and missing not at random (21/46). Multiple imputation by chained equations (MICE) was the most popular statistical method, whereas generative adversarial network-based methods and the k nearest neighbor (KNN) classification were the common deep-learning-based or traditional machine-learning-based methods, respectively. Among the 26 articles comparing the performance among medical statistical and machine learning approaches, traditional machine learning or deep learning methods generally outperformed statistical methods. Med.KNN and context-aware time-series imputation performed better for longitudinal datasets, whereas probabilistic principal component analysis and MICE-based methods were optimal for cross-sectional datasets. Conclusions: Machine learning methods show significant promise for addressing missing data in EHRs. However, no single approach provides a universally generalizable solution. Standardized benchmarking analyses are essential to evaluate these methods across different missingness scenarios.Keywords
Funding Information
- Research and Development Fund of Peking University People's Hospital (RDX2023-11)
- The National Natural Science Foundation of China (No.62102008)
- The National Natural Science Foundation of China (No. 81602939)
This publication has 67 references indexed in Scilit:
- Multiple ImputationJAMA, 2015
- Approach to Addressing Missing Data for Electronic Medical Records and Pharmacy Claims Data ResearchPharmacotherapy: The Journal of Human Pharmacology and Drug Therapy, 2015
- Imputing Missing Race/Ethnicity in Pediatric Electronic Health Records: Reducing Bias with Use of U.S. Census Location and Surname DataHealth Services Research, 2015
- Performance of the marginal structural models under various scenarios of incomplete marker's values: A simulation studyBiometrical Journal, 2014
- Evaluation of two‐fold fully conditional specification multiple imputation for longitudinal electronic health record dataStatistics in Medicine, 2014
- Missing values in deduplication of electronic patient dataJournal of the American Medical Informatics Association, 2012
- Longitudinal data analysis with non-ignorable missing dataStatistical Methods in Medical Research, 2012
- Bias and efficiency of multiple imputation compared with complete‐case analysis for missing covariate valuesStatistics in Medicine, 2010
- Comparison of techniques for handling missing covariate data within prognostic modelling studies: a simulation studyBMC Medical Research Methodology, 2010
- Multiple imputation in health‐are databases: An overview and some applicationsStatistics in Medicine, 1991