Validation of a Natural Language Processing-Assisted Chart Review Method for Identifying Advance Care Planning Documentation in Pediatric Critical Illness
Open Access
- 25 September 2026
- journal article
- research article
- Published by Wolters Kluwer Health in Critical Care Explorations
- Vol. 8 (10) , e1494
Abstract
OBJECTIVES: Reliable measurement of advance care planning (ACP) is essential for evaluating both communication practices and quality of care in pediatric critical illness. However, most ACP documentation is embedded in free-text clinical notes and cannot be captured using administrative data, while manual chart review is time-intensive and difficult to scale. We sought to validate a rule-based natural language processing (NLP) approach to identify ACP documentation in the electronic health record (EHR) compared with manual chart review. DESIGN: Retrospective cohort study. SETTING: Single-center, quaternary PICU. PATIENTS: Children younger than 21 years admitted to the PICU for greater than 24 hours following out-of-hospital cardiac arrest (OHCA) from 2012 to 2024. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: A rule-based NLP approach incorporating semi-automated chart review was developed through iterative refinement of previously validated keyword libraries to identify ACP documentation across prespecified domains in the EHR. In the validation cohort ( n = 95), NLP-assisted chart review showed complete agreement with manual chart review (F1 = 1.0) while substantially reducing abstraction time, enabling analysis of over 31,479 clinical notes in 3 days compared with 6 months for manual review of 43,179 notes. Among 125 children with OHCA, 65% had documented ACP discussions ( n = 81). Goals-of-care conversations were most common (99%), followed by limitations of life-sustaining treatment (78%), subspecialty palliative care involvement (28%), preferred location of death (9%), and hospice discussions (7%). CONCLUSIONS: NLP-assisted chart review can efficiently and accurately identify ACP documentation in pediatric critical illness for retrospective research and quality improvement efforts. This semi-automated workflow substantially reduced manual abstraction burden while maintaining excellent agreement with manual chart review, enabling effective abstraction of communication processes not captured in structured data.Keywords
This publication has 16 references indexed in Scilit:
- Society of Critical Care Medicine 2026 Guidelines on the Care and Management of Pediatric and Neonatal Intensive Care Patients at the End of LifePediatric Critical Care Medicine, 2026
- Large Language Models to Identify Advance Care Planning in Patients With Advanced CancerJournal of Pain and Symptom Management, 2024
- Forms or Free-Text? Measuring Advance Care Planning Activity Using Electronic Health RecordsJournal of Pain and Symptom Management, 2023
- Assessment of Natural Language Processing of Electronic Health Records to Measure Goals-of-Care Discussions as a Clinical Trial OutcomeJAMA Network Open, 2023
- Natural Language Processing to Identify Advance Care Planning Documentation in a Multisite Pragmatic Clinical TrialJournal of Pain and Symptom Management, 2021
- Evaluation of Use of Technologies to Facilitate Medical Chart ReviewDrug Safety, 2019
- Natural Language Processing to Assess End-of-Life Quality Indicators in Cancer Patients Receiving Palliative SurgeryJournal of Palliative Medicine, 2019
- Advance Care Planning and Parent-Reported End-of-Life Outcomes in Children, Adolescents, and Young Adults With Complex Chronic Conditions*Critical Care Medicine, 2019
- Pediatric complex chronic conditions classification system version 2: updated for ICD-10 and complex medical technology dependence and transplantationBMC Pediatrics, 2014
- Advance care planning for paediatric patientsPaediatrics & Child Health, 2008