Artificial intelligence based assessment of clinical reasoning documentation: an observational study of the impact of the clinical learning environment on resident documentation quality
Open Access
- 22 April 2025
- journal article
- research article
- Published by Springer Nature in BMC Medical Education
- Vol. 25 (1) , 1-10
Abstract
Objective measures and large datasets are needed to determine aspects of the Clinical Learning Environment (CLE) impacting the essential skill of clinical reasoning documentation. Artificial Intelligence (AI) offers a solution. Here, the authors sought to determine what aspects of the CLE might be impacting resident clinical reasoning documentation quality assessed by AI. In this observational, retrospective cross-sectional analysis of hospital admission notes from the Electronic Health Record (EHR), all categorical internal medicine (IM) residents who wrote at least one admission note during the study period July 1, 2018– June 30, 2023 at two sites of NYU Grossman School of Medicine’s IM residency program were included. Clinical reasoning documentation quality of admission notes was determined to be low or high-quality using a supervised machine learning model. From note-level data, the shift (day or night) and note index within shift (if a note was first, second, etc. within shift) were calculated. These aspects of the CLE were included as potential markers of workload, which have been shown to have a strong relationship with resident performance. Patient data was also captured, including age, sex, Charlson Comorbidity Index, and primary diagnosis. The relationship between these variables and clinical reasoning documentation quality was analyzed using generalized estimating equations accounting for resident-level clustering. Across 37,750 notes authored by 474 residents, patients who were older, had more pre-existing comorbidities, and presented with certain primary diagnoses (e.g., infectious and pulmonary conditions) were associated with higher clinical reasoning documentation quality. When controlling for these and other patient factors, variables associated with clinical reasoning documentation quality included academic year (adjusted odds ratio, aOR, for high-quality: 1.10; 95% CI 1.06–1.15; P <.001), night shift (aOR 1.21; 95% CI 1.13–1.30; P <.001), and note index (aOR 0.93; 95% CI 0.90–0.95; P <.001). AI can be used to assess complex skills such as clinical reasoning in authentic clinical notes that can help elucidate the potential impact of the CLE on resident clinical reasoning documentation quality. Future work should explore residency program and systems interventions to optimize the CLE.Keywords
All Related Versions
This publication has 58 references indexed in Scilit:
- Chatbot vs Medical Student Performance on Free-Response Clinical Reasoning ExaminationsJAMA Internal Medicine, 2023
- Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic ChallengeJAMA, 2023
- Quality and Safety Outcomes of a Hospital Merger Following a Full Integration at a Safety Net HospitalJAMA Network Open, 2022
- Spending Patterns in Region of Residency Training and Subsequent Expenditures for Care Provided by Practicing Physicians for Medicare BeneficiariesJAMA, 2014
- The Clinical Learning EnvironmentJAMA, 2013
- Types and Origins of Diagnostic Errors in Primary Care SettingsJAMA Internal Medicine, 2013
- Evaluating Obstetrical Residency Programs Using Patient OutcomesJAMA, 2009
- Association of Workload of On-Call Medical Interns With On-Call Sleep Duration, Shift Duration, and Participation in Educational ActivitiesJAMA, 2008
- Evaluating Medical Training Programs by the Quality of Care Delivered by Their AlumniJAMA, 2007
- House Staff Team Workload and Organization Effects on Patient Outcomes in an Academic General Internal Medicine Inpatient ServiceArchives of internal medicine (1960), 2007