Chenyu Li

Symbolic clinical natural language processing (NLP) systems remain widely used for extracting clinical concepts from electronic health record (EHR) narratives, but maintaining rule resources requires extensive manual error analysis and rule refinement. This study investigates whether large language models (LLMs) can assist in identifying extraction errors and generating candidate rules to improve symbolic clinical NLP systems. Using error reports derived from a multi-site evaluation of a previously validated symbolic model for cognitive and neuropsychiatric-related clinical concepts, we developed a human-in-the-loop framework, REFINE. The framework first uses LLMs to classify extraction errors and generate explanatory reasoning, which can then be incorporated into prompts for rule generation. Three LLMs (GPT-5.2, GPT-4o, GPT-4o-mini) were evaluated under four prompting conditions. LLM-generated rule sets improved performance compared with the baseline NLP-CAM system, increasing F1-score from 0.37 to 0.58. These findings suggest that LLMs can support scalable rule refinement for symbolic clinical NLP systems.

Introduction: The increasing aging population raises significant concerns about the ability of individuals to age healthily, avoiding chronic diseases and maintaining cognitive and physical functions. However, the pathways through which SDOH factors are associated with healthy aging remain unclear. Methods: This retrospective cohort study uses the registered tier data from the All of Us Research Program (AoURP) registered tier dataset v7. Eligible study participants are those aged 50 and older who have responded to any of the SDOH survey questions with available EHR data. Three different algorithms were trained (logistic regression [LR], multi-layer perceptron [MLP], and extreme gradient boosting [XGBoost]). The outcome is healthy aging, which is measured by a composite score of the status for 1) comorbidities, 2) cognitive conditions, and 3) mobility function. We evaluate the model performance by area under the receiver operating characteristic curve (AUROC) and assess the fairness of best-performed model through predictive parity. Feature importance is analyzed using SHapley Additive exPlanations (SHAP) values. Results: Our study included 99,935 participants aged 50 and above, and the mean (SD) age was 74 (9.3), with 55,294 (55.3%) females, 67,457 (67.5%) Whites, 11,109 (11.1%) Hispanic ethnicity, and 44,109 (44.1%) are classified as healthy aging. Most of the individuals lived in their own house (64%), were married (51%), obtained college or advanced degrees (74%), and had Medicare (56.2%). The best predictive model was XGBoost with random oversampler, with a performance of AUROC [95% CI]: 0.793 [0.788–0.796], F1 score: 0.697 [0.692–0.701], recall: 0.739 [0.732–0.748], precision: 0.659 [0.655–0.663], and accuracy: 0.716 [0.712–0.720], and the XGBoost model achieved predictive parity by similar positive and negative predictive values across race and sex groups (0.86–1.06). In feature importance analysis, health insurance type is ranked as the most predictive feature, followed by employment status, substance use, and health insurance coverage (yes/no). Conclusion: In this cohort study, XGBoost model accurately predicted individuals achieving healthy aging, outperforming LR and MLP. Our findings underscore the significant role of health insurance in contributing to healthy aging.

Background: Delirium is often underdiagnosed in clinical practice and is not routinely coded for billing. While manual chart review can identify delirium, it is labor-intensive and impractical for large-scale studies. Natural language processing (NLP) can analyze unstructured text in electronic health records (EHRs) to extract meaningful clinical information. Methods: To support national integration of NLP for EHR-based delirium identification across different institutions, we launched the Delirium Interest Group within the national Evolve to Next-Gen Accrual to Clinical Trials (ENACT) NLP Working Group. This paper outlines our initial efforts to standardize, evaluate, and translate an NLP-based delirium detection model into the i2b2/ENACT platform. Results: Multisite contextual inquiry identified several key challenges, including variations in local screening practices (eg, tools used, documentation frequency, and quality control), the need for harmonized definitions in the context of EHRs, and the complexity of modeling temporal logic. Multisite NLP evaluation revealed variable performance degradation driven by differences in delirium screening practices, clinical documentation patterns and semantics, and note syntactic structures. Conclusion: Our work represents an important first step toward enabling scalable and standardized NLP-based delirium detection across institutions. By engaging diverse institutions through the ENACT NLP Working Group, we identified shared challenges and site-specific variations that impact model implementation and performance. Our collaborative approach enabled the development of a more robust framework for delirium identification across heterogeneous EHR systems. Future efforts will build on this foundation to enhance the validity, usability, and translational impact of delirium detection.

Background: Transportation insecurity is a known barrier to accessing eye care and is associated with poorer visual outcomes for patients. However, its mention is seldom captured in structured data fields in electronic health records, limiting efforts to identify and support affected patients. Free-text clinical documentation may more efficiently capture information on transportation-related challenges than structured data. Objective: In this study, we aimed to identify mention of transportation insecurity in free-text ophthalmology clinic notes using natural language processing (NLP). Methods: In this retrospective, cross-sectional study, we examined ophthalmology clinic notes of adult patients with an encounter at a tertiary academic eye center from 2016 to 2023. Demographic information and free text from clinical notes were extracted from electronic health records and deidentified for analysis. Free text was used to develop a rule-based NLP algorithm to identify transportation insecurity. The NLP algorithm was trained and validated using a gold-standard expert review, and precision, recall, and F₁-scores were used to evaluate the algorithm’s performance. Logistic regression evaluated associations between demographics and transportation insecurity. Results: A total of 1,801,572 clinical notes of 118,518 unique patients were examined, and the NLP algorithm identified 726 (0.6%) patients with transportation insecurity. The algorithm’s precision, recall, and F₁-score were 0.860, 0.960, and 0.778, respectively, indicating high agreement with the gold-standard expert review. Patients with identified transportation insecurity were more likely to be older (OR 3.01, 95% CI 2.38‐3.78 for those aged ≥80 vs 18‐60 y) and less likely to identify as Asian (OR 0.04, 95% CI 0‐0.18 for Asian patients vs White patients). There was no difference by sex (OR 1.13, 95% CI 0.97‐1.31) or between the Black and White races (OR 0.98, 95% CI 0.79‐1.22). Conclusions: NLP has the potential to identify patients experiencing transportation insecurity from ophthalmology clinic notes, which may help to facilitate referrals to transportation resources.

Objective: Electronic Health Record (EHR) data are critical for advancing translational research and AI technologies. The ENACT network offers access to structured EHR data across 57 CTSA hubs. However, substantial information is contained in clinical narratives, requiring natural language processing (NLP) for research. The ENACT NLP Working Group was formed to make NLP-derived clinical information accessible and queryable across the network. Methods: We established the ENACT NLP Working Group with 13 sites selected based on criteria including clinical notes access, IT infrastructure, NLP expertise, and institutional support. We divided sites into five focus groups targeting clinical tasks within disease contexts. Each focus group consisted of two development sites and two validation sites. We extended the ENACT ontology to standardize NLP-derived data and conducted multisite evaluations using the Open Health Natural Language Processing (OHNLP) Toolkit. Results: The working group achieved 100% site retention and deployed NLP infrastructure across all sites. We developed and validated NLP algorithms for rare disease phenotyping, social determinants of health, opioid use disorder, sleep phenotyping, and delirium phenotyping. Performance varied across sites (F1 scores 0.53–0.96), highlighting data heterogeneity impacts. We extended the ENACT common data model and ontology to incorporate NLP-derived data while maintaining Shared Health Research Informatics NEtwork (SHRINE) compatibility. Conclusion: This demonstrates feasibility of deploying NLP infrastructure across large, federated networks. The focus group approach proved more practical than general-purpose approaches. Key lessons include the challenge of data heterogeneity and importance of collaborative governance. This work also provides a foundation that other networks can build on to implement NLP capabilities for translational research.

Introduction: The increasing aging population raises significant concerns about the ability of individuals to age healthily, avoiding chronic diseases and maintaining cognitive and physical functions. However, the pathways through which SDOH factors are associated with healthy aging remain unclear. Methods: This retrospective cohort study used the registered tier data from the All of Us Research Program (AoURP) registered tier dataset v7. Eligible study participants are those aged 50 and older who have responded to any of the SDOH survey questions with available EHR data. Three different algorithms were trained (logistic regression [LR], multi-layer perceptron [MLP], and extreme gradient boosting [XGBoost]). The outcome is healthy aging, which is measured by a composite score of the status for 1) comorbidities, 2) cognitive conditions, and 3) mobility function. We evaluate the model performance by area under the receiver operating characteristic curve (AUROC) and assess the fairness of best-performed model through predictive parity. Feature importance is analyzed using SHapley Additive exPlanations (SHAP) values. Results: Our study included 99,935 participants aged 50 and above, and the mean (SD) age was 74 (9.3), with 55,294 (55.3%) females, 67,457 (67.5%) Whites, 11,109 (11.1%) Hispanic ethnicity, and 44,109 (44.1%) are classified as healthy aging. Most of the individuals lived in their own house (64%), were married (51%), obtained college or advanced degrees (74%), and had Medicare (56.2%). The best predictive model was XGBoost with random oversampler, with a performance of AUROC [95% CI]: 0.793 [0.788-0.796], F1 score: 0.697 [0.692-0.701], recall: 0.739 [0.732-0.748], precision: 0.659 [0.655-0.663], and accuracy: 0.716 [0.712-0.720], and the XGBoost model achieved predictive parity by similar positive and negative predictive values across race and sex groups (0.86-1.06). In feature importance analysis, health insurance type is ranked as the most predictive feature, followed by employment status, substance use, and health insurance coverage (yes/no). Conclusion: In this cohort study, XGBoost model accurately predicted individuals achieving healthy aging, outperforming LR and MLP. Our findings underscore the significant role of health insurance in contributing to healthy aging.

Background: Social determinants of health (SDoH), such as socioeconomics and neighborhoods, strongly influence health outcomes. However, the current state of standardized SDoH data in electronic health records (EHRs) is lacking, a significant barrier to research and care quality.Methods: We conducted a PubMed search using “SDOH” and “EHR” Medical Subject Headings terms, analyzing included articles across five domains: 1) SDoH screening and assessment approaches, 2) SDoH data collection and documentation, 3) Use of natural language processing (NLP) for extracting SDoH, 4) SDoH data and health outcomes, and 5) SDoH-driven interventions.Results: Of 685 articles identified, 324 underwent full review. Key findings include implementation of tailored screening instruments, census and claims data linkage for contextual SDoH profiles, NLP systems extracting SDoH from notes, associations between SDoH and healthcare utilization and chronic disease control, and integrated care management programs. However, variability across data sources, tools, and outcomes underscores the need for standardization.Discussion: Despite progress in identifying patient social needs, further development of standards, predictive models, and coordinated interventions is critical for SDoH-EHR integration. Additional database searches could strengthen this scoping review. Ultimately, widespread capture, analysis, and translation of multidimensional SDoH data into clinical care is essential for promoting health equity.

Background: Error analysis plays a crucial role in clinical concept extraction, a fundamental subtask within clinical natural language processing (NLP). The process typically involves a manual review of error types, such as contextual and linguistic factors contributing to their occurrence, and the identification of underlying causes to refine the NLP model and improve its performance. Conducting error analysis can be complex, requiring a combination of NLP expertise and domain-specific knowledge. Due to the high heterogeneity of electronic health record (EHR) settings across different institutions, challenges may arise when attempting to standardize and reproduce the error analysis process. Objectives: This study aims to facilitate a collaborative effort to establish common definitions and taxonomies for capturing diverse error types, fostering community consensus on error analysis for clinical concept extraction tasks. Materials and Methods: We iteratively developed and evaluated an error taxonomy based on existing literature, standards, real-world data, multisite case evaluations, and community feedback. The finalized taxonomy was released in both .dtd and .owl formats at the Open Health Natural Language Processing Consortium. The taxonomy is compatible with several different open-source annotation tools, including MAE, Brat, and MedTator. Results: The resulting error taxonomy comprises 43 distinct error classes, organized into 6 error dimensions and 4 properties, including model type (symbolic and statistical machine learning), evaluation subject (model and human), evaluation level (patient, document, sentence, and concept), and annotation examples. Internal and external evaluations revealed strong variations in error types across methodological approaches, tasks, and EHR settings. Key points emerged from community feedback, including the need to enhancing clarity, generalizability, and usability of the taxonomy, along with dissemination strategies. Conclusion: The proposed taxonomy can facilitate the acceleration and standardization of the error analysis process in multi-site settings, thus improving the provenance, interpretability, and portability of NLP models. Future researchers could explore the potential direction of developing automated or semi-automated methods to assist in the classification and standardization of error analysis.