Medical Data Pecking: A Context-Aware Approach for Automated Quality Evaluation of Structured Medical Data
Authors:
- Atai AmbusAAAtai Ambus
- Ran Gilad-BachrachRGRan Gilad-Bachrach
Preprint
Abstract
Background: The use of Electronic Health Records (EHRs) for epidemiological studies and artificial intelligence (AI) training is increasing rapidly. The reliability of the results depends on the accuracy and completeness of EHR data. However, EHR data often contain significant quality issues, including misrepresentations of subpopulations, biases, and systematic errors, as they are primarily collected for clinical and billing purposes. Existing quality assessment methods remain insufficient, lacking systematic procedures to assess data fitness for research. Methods: We present the Medical Data Pecking approach, which adapts unit testing and coverage concepts from software engineering to identify data quality concerns. We demonstrate our approach using the Medical Data Pecking Tool (MDPT), which consists of two main components: (1) an automated test generator that uses large language models and grounding techniques to create a test suite from data and study descriptions, and (2) a data testing framework that executes these tests, reporting potential errors and coverage. Results: We evaluated MDPT on three datasets: All of Us (AoU), MIMIC-III, and SyntheticMass, generating 55-73 tests per cohort across four conditions. These tests correctly identified 20-43 non-aligned or non-conforming data issues. We present a detailed analysis of the LLM-generated test suites in terms of reference grounding and value accuracy. Conclusion: Our approach incorporates external medical knowledge to enable context-sensitive data quality testing as part of the data analysis workflow to improve the validity of its outcomes. Our approach tackles these challenges from a quality assurance perspective, laying the foundation for further development such as additional data modalities and improved grounding methods.Keywords
All Related Versions
This publication has 41 references indexed in Scilit:
- Artificial Intelligence Algorithm for Subclinical Breast Cancer DetectionJAMA Network Open, 2024
- Breast Cancer in the Ashkenazi Jewish PopulationCurrent Breast Cancer Reports, 2024
- OHDSI Standardized Vocabularies—a large-scale centralized reference ontology for international data harmonizationJournal of the American Medical Informatics Association, 2024
- Zero-shot interpretable phenotyping of postpartum hemorrhage using large language modelsnpj Digital Medicine, 2023
- Automating Electronic Health Record Data Quality AssessmentJournal of Medical Systems, 2023
- Increasing trust in real-world evidence through evaluation of observational data qualityJournal of the American Medical Informatics Association, 2021
- Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care recordJournal of the American Medical Informatics Association, 2017
- Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and PracticesIEEE Access, 2017
- Code coverage and test suite effectiveness: Empirical study with real bugs in large systems2015
- The Economics of Unit TestingEmpirical Software Engineering, 2006