Considerations for the Use of Machine Learning Extracted Real-World Data to Support Evidence Generation: A Research-Centric Evaluation Framework
Open Access
Abstract
A vast amount of real-world data, such as pathology reports and clinical notes, are captured as unstructured text in electronic health records (EHRs). However, this information is both difficult and costly to extract through human abstraction, especially when scaling to large datasets is needed. Fortunately, Natural Language Processing (NLP) and Machine Learning (ML) techniques provide promising solutions for a variety of information extraction tasks such as identifying a group of patients who have a specific diagnosis, share common characteristics, or show progression of a disease. However, using these ML-extracted data for research still introduces unique challenges in assessing validity and generalizability to different cohorts of interest. In order to enable effective and accurate use of ML-extracted real-world data (RWD) to support research and real-world evidence generation, we propose a research-centric evaluation framework for model developers, ML-extracted data users and other RWD stakeholders. This framework covers the fundamentals of evaluating RWD produced using ML methods to maximize the use of EHR data for research purposes.Keywords
Funding Information
- Flatiron Health
This publication has 35 references indexed in Scilit:
- Clinical information extraction applications: A literature reviewJournal of Biomedical Informatics, 2018
- Harnessing the Power of Real‐World Evidence (RWE): A Checklist to Ensure Regulatory‐Grade Data QualityClinical Pharmacology & Therapeutics, 2017
- DeepPhe: A Natural Language Processing System for Extracting Cancer Phenotypes from Clinical RecordsCancer Research, 2017
- Assessment of the need for separate test set and number of medical images necessary for deep learning: a sub-sampling studybioRxiv, 2017
- Natural language processing systems for capturing and standardizing unstructured clinical information: A systematic reviewJournal of Biomedical Informatics, 2017
- Natural Language Processing in OncologyJAMA Oncology, 2016
- Opportunities and Challenges in Leveraging Electronic Health Record Data in OncologyFuture Oncology, 2016
- Good practices for quantitative bias analysisInternational Journal of Epidemiology, 2014
- Predicting sample size required for classification performanceBMC Medical Informatics and Decision Making, 2012
- The analysis of survival data: the Kaplan–Meier methodKidney International, 2008