David Chartash
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Research Interests
David Chartash has a total of 89 co-authors. Here are the top co-authors by number of shared publications:
- Conrad W. Safranek
Department of Emergency Medicine, Yale School of Medicine, 464 Congress Avenue, #260, New Haven, CT, 06519, United States, 1 203-737-7694
12 shared publications - Richard Andrew Taylor
Department of Emergency Medicine University of Virginia Charlottesville Virginia USA
12 shared publications - Aidan Gilson
Section for Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, CT, United States
11 shared publications - Vimig Socrates
Department of Biomedical Informatics and Data Science, School of Medicine, Yale University, New Haven, CT, United States
11 shared publications - Ling Chi
Section for Biomedical Informatics and Data Science, Yale University School of Medicine, New Haven, CT, United States
9 shared publications - Thomas Huang
School of Medicine, Yale University, New Haven, CT, United States
9 shared publications
Latest Publications
- Open Access
Normative Temperature, Demographic Categories, and the Limits of Infrared Thermometry
Objective: Temperature readings are routinely stratified by race and ethnicity, though significant group differences may reflect physiology, instrument noise, or categorization rather than biology. This study tests whether these categories partition facial infrared thermography measurements beyond individual variation and baseline device noise. Approach: A secondary analysis of thermography data ( n = 951 adults) compares inner-canthus maximum ( T CEmax ) and full-face maximum ( T max ) temperatures across five administrative categories via one-way analysis of variance (ANOVA), mixed-effects modeling, Bland–Altman analysis against oral reference, and Monte Carlo sensitivity analysis. Main results: ANOVA returns significant group differences ( p < 0.001; η 2 = 0.04 for T CEmax , 0.03 for T max ), but the total spread (0.2 °C to 0.3 °C) is comparable to device accuracy and within-subject variability. Individual variation accounts for 95% of T CEmax variance, and adding simulated Gaussian noise to participant-level temperatures ( σ = 0.10– 0.30 °C) preserves the complete group-mean rank order in as few as 37.5% of iterations. Significance: Testing whether a group difference survives perturbation at the scale of instrument accuracy offers a check against treating significance alone as evidence of physiology. Here, observed group differences are small relative to measurement variability, and their physiological interpretation remains uncertain, with implications for clinical tools that treat administrative categories as biologically meaningful.
Open AccessNormative Temperature, Demographic Categories, and the Limits of Infrared Thermometry
Objective: Temperature readings are routinely stratified by race and ethnicity, though significant group differences may reflect physiology, instrument noise, or categorization rather than biology. This study tests whether these categories partition facial infrared thermography measurements beyond individual variation and baseline device noise. Approach: A secondary analysis of thermography data ( n = 951 adults) compares inner-canthus maximum ( T CEmax ) and full-face maximum ( T max ) temperatures across five administrative categories via one-way analysis of variance (ANOVA), mixed-effects modeling, Bland–Altman analysis against oral reference, and Monte Carlo sensitivity analysis. Main results: ANOVA returns significant group differences ( p < 0.001; η 2 = 0.04 for T CEmax , 0.03 for T max ), but the total spread (0.2 °C to 0.3 °C) is comparable to device accuracy and within-subject variability. Individual variation accounts for 95% of T CEmax variance, and adding simulated Gaussian noise to participant-level temperatures ( σ = 0.10– 0.30 °C) preserves the complete group-mean rank order in as few as 37.5% of iterations. Significance: Testing whether a group difference survives perturbation at the scale of instrument accuracy offers a check against treating significance alone as evidence of physiology. Here, observed group differences are small relative to measurement variability, and their physiological interpretation remains uncertain, with implications for clinical tools that treat administrative categories as biologically meaningful.
- Open Access
- 28 July 2026
- Published by MDPI AG in International Medical Education
- Vol. 5 (3) , 71
Feedback and Feedback Culture in Medical Education: Evolution, Evidence, and Emerging Horizons
Feedback is a key component of medical education and is essential for progression throughout medical training. While extensive literature describes best practices for delivering feedback, less is known about the critical role of feedback culture and its impact on feedback practices. This narrative review aims to discuss the evolution of feedback in medicine, the role of feedback culture and its impact on feedback practices, strategies to foster a positive feedback culture, and opportunities and challenges for feedback in the age of artificial intelligence. Relevant articles were identified through database searches and were selected based on relevance to the aims of this review. Our review demonstrates that the literature has shifted to emphasize the importance of broader sociocultural factors in shaping feedback and feedback interactions. Determinants of a positive feedback culture stem from all levels of feedback interactions, including individual learner’s engagement with feedback, interpersonal relationships and strong educational alliances, and institutional level factors that generate a positive culture. These determinants may then be translated into strategies to enhance and sustain a positive feedback culture. The advent of artificial intelligence in medical education has provided opportunities to enhance feedback, although the impact on feedback culture remains unknown. Future research must focus on effective strategies to measure feedback culture which must then be utilized to evaluate the impact of feedback culture on trainee and patient outcomes.
- 28 July 2026
- Published by MDPI AG in International Medical Education
- Vol. 5 (3) , 71
Open AccessFeedback and Feedback Culture in Medical Education: Evolution, Evidence, and Emerging Horizons
Feedback is a key component of medical education and is essential for progression throughout medical training. While extensive literature describes best practices for delivering feedback, less is known about the critical role of feedback culture and its impact on feedback practices. This narrative review aims to discuss the evolution of feedback in medicine, the role of feedback culture and its impact on feedback practices, strategies to foster a positive feedback culture, and opportunities and challenges for feedback in the age of artificial intelligence. Relevant articles were identified through database searches and were selected based on relevance to the aims of this review. Our review demonstrates that the literature has shifted to emphasize the importance of broader sociocultural factors in shaping feedback and feedback interactions. Determinants of a positive feedback culture stem from all levels of feedback interactions, including individual learner’s engagement with feedback, interpersonal relationships and strong educational alliances, and institutional level factors that generate a positive culture. These determinants may then be translated into strategies to enhance and sustain a positive feedback culture. The advent of artificial intelligence in medical education has provided opportunities to enhance feedback, although the impact on feedback culture remains unknown. Future research must focus on effective strategies to measure feedback culture which must then be utilized to evaluate the impact of feedback culture on trainee and patient outcomes.
- Open Access
- 28 May 2026
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 12 (1) , e89996
Ambient AI Scribes to Create Educational Feedback Notes for Medical Students: Randomized Trial
Background: High-quality observation and feedback contribute to the development of clinical competence and professional growth in medical education. Faculty often struggle to translate verbal observations into written feedback because of documentation burden and competing demands. Ambient artificial intelligence (AI) scribes, already adopted in clinical practice, may address this challenge by capturing verbal exchanges and generating structured notes. Objective: The purpose of this study was to examine the use of ambient AI scribes to generate educational feedback notes during a formative medical interviewing workshop for first-year medical students in March and April 2025. Methods: Thirteen instructors were randomized to control (human-only) or intervention (AI scribe–assisted) workflows to complete narrative feedback forms. The intervention group used an AI scribe to generate transcripts of student-instructor encounters, which were then summarized into feedback notes using a large language model and edited by the instructors before submission. All narratives were scored using the Evaluation of Feedback Captured Tool (EFeCT). Factual accuracy of a subsample of unedited AI feedback summaries was reviewed against the source transcripts. Task load and usability were measured using NASA Task Load Index and System Usability Scale, respectively. Results: Instructors submitted feedback on 92.2% (94/102) of the students. EFeCT scores on the scale from 0 to 5 were higher for human-edited AI narratives (median 3.00, IQR 2.00-4.00) and unedited AI summaries (median 3.00, IQR 3.00-4.00) than for human-only narratives (median 2.00, IQR 1.75-3.00; P <.001). Human-only narratives were shorter than AI-assisted outputs ( P <.001). Review of 117 AI-generated feedback elements showed a 6.8% (n=8) mischaracterization and 1.7% (n=2) hallucination rate, with most errors corrected during editing. Task load was high, and usability was marginal in both the control and intervention groups, with no significant differences ( P =.31 and P =.40, respectively). Conclusions: An ambient AI scribe–assisted workflow improved the quality of written narrative feedback with no observed increase in instructor effort compared to human-only documentation. Although occasional inaccuracies required review, this innovation has the potential to transform feedback documentation.
- 28 May 2026
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 12 (1) , e89996
Open AccessAmbient AI Scribes to Create Educational Feedback Notes for Medical Students: Randomized Trial
Background: High-quality observation and feedback contribute to the development of clinical competence and professional growth in medical education. Faculty often struggle to translate verbal observations into written feedback because of documentation burden and competing demands. Ambient artificial intelligence (AI) scribes, already adopted in clinical practice, may address this challenge by capturing verbal exchanges and generating structured notes. Objective: The purpose of this study was to examine the use of ambient AI scribes to generate educational feedback notes during a formative medical interviewing workshop for first-year medical students in March and April 2025. Methods: Thirteen instructors were randomized to control (human-only) or intervention (AI scribe–assisted) workflows to complete narrative feedback forms. The intervention group used an AI scribe to generate transcripts of student-instructor encounters, which were then summarized into feedback notes using a large language model and edited by the instructors before submission. All narratives were scored using the Evaluation of Feedback Captured Tool (EFeCT). Factual accuracy of a subsample of unedited AI feedback summaries was reviewed against the source transcripts. Task load and usability were measured using NASA Task Load Index and System Usability Scale, respectively. Results: Instructors submitted feedback on 92.2% (94/102) of the students. EFeCT scores on the scale from 0 to 5 were higher for human-edited AI narratives (median 3.00, IQR 2.00-4.00) and unedited AI summaries (median 3.00, IQR 3.00-4.00) than for human-only narratives (median 2.00, IQR 1.75-3.00; P <.001). Human-only narratives were shorter than AI-assisted outputs ( P <.001). Review of 117 AI-generated feedback elements showed a 6.8% (n=8) mischaracterization and 1.7% (n=2) hallucination rate, with most errors corrected during editing. Task load was high, and usability was marginal in both the control and intervention groups, with no significant differences ( P =.31 and P =.40, respectively). Conclusions: An ambient AI scribe–assisted workflow improved the quality of written narrative feedback with no observed increase in instructor effort compared to human-only documentation. Although occasional inaccuracies required review, this innovation has the potential to transform feedback documentation.
- 22 March 2026
- Published by Elsevier in Patient Education and Counseling
- Vol. 149, 109597
From complexity to clarity: GPT-4 for patient-centered prostate cancer pathology reports
- 22 March 2026
- Published by Elsevier in Patient Education and Counseling
- Vol. 149, 109597
From complexity to clarity: GPT-4 for patient-centered prostate cancer pathology reports
- Open Access
- 24 February 2026
- Published by Springer Nature in BMC Medical Informatics and Decision Making
- Vol. 26 (1)
Simulated evaluation of large language model stepwise diagnostic reasoning with real-world chest pain encounters and Bayesian networks
Real-world evaluation of large language models (LLMs) as clinical diagnostic aids is limited by the reliance on static vignettes and retrospective data, wh
- 24 February 2026
- Published by Springer Nature in BMC Medical Informatics and Decision Making
- Vol. 26 (1)
Open AccessSimulated evaluation of large language model stepwise diagnostic reasoning with real-world chest pain encounters and Bayesian networks
Real-world evaluation of large language models (LLMs) as clinical diagnostic aids is limited by the reliance on static vignettes and retrospective data, wh
- 12 December 2025
- Published by Springer Nature in Advances in Health Sciences Education
- Vol. 31 (4) , 1-26
Can you really infer that? An exploratory study using the reasoning task typology to analyze clinic notes
Assessing clinical reasoning is a challenge in all stages of medical education. Beyond the verbal communication of clinical reasoning within the hierarchy of medical education, written communication is a multi-purpose complex adjunct to support the educational process. This study applied and validated a retrospective template analysis of clinical notes using a comprehensive clinical reasoning typology intended to broaden the measurement of clinical reasoning beyond predominantly cognitive activities. Applying the typology to written communication demonstrates that increasing complexity of the analysis is mediated by intra-textual and discourse coherence, as well as the semantic ambiguity of clinical writing. The findings from this study suggest that a significant amount of information during the clinical encounter is pertinent to clinical reasoning but inferred within the contextual medical knowledge held by the writing and reading physician. Ultimately the structure and purpose of the clinical narrative reinforces both a physician-relative and culturally shaped practice of documentation. Beyond specific findings from this study, as the electronic medical record further matures and is shaped by both technology and the computer itself in the exam room, educators must seek to teach the task of clinical reasoning explication through writing, yet pay attention to the technology within the encounter and medical record itself, which shapes the clinical narrative.
- 12 December 2025
- Published by Springer Nature in Advances in Health Sciences Education
- Vol. 31 (4) , 1-26
Can you really infer that? An exploratory study using the reasoning task typology to analyze clinic notes
Assessing clinical reasoning is a challenge in all stages of medical education. Beyond the verbal communication of clinical reasoning within the hierarchy of medical education, written communication is a multi-purpose complex adjunct to support the educational process. This study applied and validated a retrospective template analysis of clinical notes using a comprehensive clinical reasoning typology intended to broaden the measurement of clinical reasoning beyond predominantly cognitive activities. Applying the typology to written communication demonstrates that increasing complexity of the analysis is mediated by intra-textual and discourse coherence, as well as the semantic ambiguity of clinical writing. The findings from this study suggest that a significant amount of information during the clinical encounter is pertinent to clinical reasoning but inferred within the contextual medical knowledge held by the writing and reading physician. Ultimately the structure and purpose of the clinical narrative reinforces both a physician-relative and culturally shaped practice of documentation. Beyond specific findings from this study, as the electronic medical record further matures and is shaped by both technology and the computer itself in the exam room, educators must seek to teach the task of clinical reasoning explication through writing, yet pay attention to the technology within the encounter and medical record itself, which shapes the clinical narrative.
- Open Access
- 7 July 2025
- Published by American Academy of Child and Adolescent Psychiatry in JAACAP Connect
- Vol. 13 (1)
Harnessing Large Language Models to Promote Attainment of ACGME Competencies in Child Psychiatry
Artificial intelligence and large language models (LLMs) such as ChatGPT are spurring both enthusiasm and trepidation within the medical education community. Here, we explore the application of LLMs in medical education in child and adolescent psychiatry in pursuit of the Accreditation Council for Graduate Medical Education (ACGME) Program Requirements for Graduate Medical Education. In particular, we demonstrate how one might leverage LLMs to achieve ACGME competencies in clinical skills in the major treatment modalities. We use the voice feature of the ChatGPT app to simulate a patient interaction during which the provider must engage in crisis intervention. We discuss the realistic nature of the scenario produced by ChatGPT, the ability of ChatGPT to serve as a clinical coach, and the opportunities afforded by prompt engineering. Interacting with LLMs throughout medical education in child psychiatry both promotes innovation and fosters an understanding of their limitations in clinical care. Fluency in the language of these models is of the utmost importance as their influence in health care continues to expand.
- 7 July 2025
- Published by American Academy of Child and Adolescent Psychiatry in JAACAP Connect
- Vol. 13 (1)
Open AccessHarnessing Large Language Models to Promote Attainment of ACGME Competencies in Child Psychiatry
Artificial intelligence and large language models (LLMs) such as ChatGPT are spurring both enthusiasm and trepidation within the medical education community. Here, we explore the application of LLMs in medical education in child and adolescent psychiatry in pursuit of the Accreditation Council for Graduate Medical Education (ACGME) Program Requirements for Graduate Medical Education. In particular, we demonstrate how one might leverage LLMs to achieve ACGME competencies in clinical skills in the major treatment modalities. We use the voice feature of the ChatGPT app to simulate a patient interaction during which the provider must engage in crisis intervention. We discuss the realistic nature of the scenario produced by ChatGPT, the ability of ChatGPT to serve as a clinical coach, and the opportunities afforded by prompt engineering. Interacting with LLMs throughout medical education in child psychiatry both promotes innovation and fosters an understanding of their limitations in clinical care. Fluency in the language of these models is of the utmost importance as their influence in health care continues to expand.
- Open Access
- 11 April 2025
- Published by JMIR Publications Inc. in JMIR Aging
- Vol. 8, e69504
Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study
Background: Polypharmacy, the concurrent use of multiple medications, is prevalent among older adults and associated with increased risks for adverse drug events including falls. Deprescribing, the systematic process of discontinuing potentially inappropriate medications, aims to mitigate these risks. However, the practical application of deprescribing criteria in emergency settings remains limited due to time constraints and criteria complexity. Objective: This study aims to evaluate the performance of a large language model (LLM)–based pipeline in identifying deprescribing opportunities for older emergency department (ED) patients with polypharmacy, using 3 different sets of criteria: Beers, Screening Tool of Older People’s Prescriptions, and Geriatric Emergency Medication Safety Recommendations. The study further evaluates LLM confidence calibration and its ability to improve recommendation performance. Methods: We conducted a retrospective cohort study of older adults presenting to an ED in a large academic medical center in the Northeast United States from January 2022 to March 2022. A random sample of 100 patients (712 total oral medications) was selected for detailed analysis. The LLM pipeline consisted of two steps: (1) filtering high-yield deprescribing criteria based on patients’ medication lists, and (2) applying these criteria using both structured and unstructured patient data to recommend deprescribing. Model performance was assessed by comparing model recommendations to those of trained medical students, with discrepancies adjudicated by board-certified ED physicians. Selective prediction, a method that allows a model to abstain from low-confidence predictions to improve overall reliability, was applied to assess the model’s confidence and decision-making thresholds. Results: The LLM was significantly more effective in identifying deprescribing criteria (positive predictive value: 0.83; negative predictive value: 0.93; McNemar test for paired proportions: χ21=5.985; P=.02) relative to medical students, but showed limitations in making specific deprescribing recommendations (positive predictive value=0.47; negative predictive value=0.93). Adjudication revealed that while the model excelled at identifying when there was a deprescribing criterion related to one of the patient’s medications, it often struggled with determining whether that criterion applied to the specific case due to complex inclusion and exclusion criteria (54.5% of errors) and ambiguous clinical contexts (eg, missing information; 39.3% of errors). Selective prediction only marginally improved LLM performance due to poorly calibrated confidence estimates. Conclusions: This study highlights the potential of LLMs to support deprescribing decisions in the ED by effectively filtering relevant criteria. However, challenges remain in applying these criteria to complex clinical scenarios, as the LLM demonstrated poor performance on more intricate decision-making tasks, with its reported confidence often failing to align with its actual success in these cases. The findings underscore the need for clearer deprescribing guidelines, improved LLM calibration for real-world use, and better integration of human–artificial intelligence workflows to balance artificial intelligence recommendations with clinician judgment.
- 11 April 2025
- Published by JMIR Publications Inc. in JMIR Aging
- Vol. 8, e69504
Open AccessIdentifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study
Background: Polypharmacy, the concurrent use of multiple medications, is prevalent among older adults and associated with increased risks for adverse drug events including falls. Deprescribing, the systematic process of discontinuing potentially inappropriate medications, aims to mitigate these risks. However, the practical application of deprescribing criteria in emergency settings remains limited due to time constraints and criteria complexity. Objective: This study aims to evaluate the performance of a large language model (LLM)–based pipeline in identifying deprescribing opportunities for older emergency department (ED) patients with polypharmacy, using 3 different sets of criteria: Beers, Screening Tool of Older People’s Prescriptions, and Geriatric Emergency Medication Safety Recommendations. The study further evaluates LLM confidence calibration and its ability to improve recommendation performance. Methods: We conducted a retrospective cohort study of older adults presenting to an ED in a large academic medical center in the Northeast United States from January 2022 to March 2022. A random sample of 100 patients (712 total oral medications) was selected for detailed analysis. The LLM pipeline consisted of two steps: (1) filtering high-yield deprescribing criteria based on patients’ medication lists, and (2) applying these criteria using both structured and unstructured patient data to recommend deprescribing. Model performance was assessed by comparing model recommendations to those of trained medical students, with discrepancies adjudicated by board-certified ED physicians. Selective prediction, a method that allows a model to abstain from low-confidence predictions to improve overall reliability, was applied to assess the model’s confidence and decision-making thresholds. Results: The LLM was significantly more effective in identifying deprescribing criteria (positive predictive value: 0.83; negative predictive value: 0.93; McNemar test for paired proportions: χ21=5.985; P=.02) relative to medical students, but showed limitations in making specific deprescribing recommendations (positive predictive value=0.47; negative predictive value=0.93). Adjudication revealed that while the model excelled at identifying when there was a deprescribing criterion related to one of the patient’s medications, it often struggled with determining whether that criterion applied to the specific case due to complex inclusion and exclusion criteria (54.5% of errors) and ambiguous clinical contexts (eg, missing information; 39.3% of errors). Selective prediction only marginally improved LLM performance due to poorly calibrated confidence estimates. Conclusions: This study highlights the potential of LLMs to support deprescribing decisions in the ED by effectively filtering relevant criteria. However, challenges remain in applying these criteria to complex clinical scenarios, as the LLM demonstrated poor performance on more intricate decision-making tasks, with its reported confidence often failing to align with its actual success in these cases. The findings underscore the need for clearer deprescribing guidelines, improved LLM calibration for real-world use, and better integration of human–artificial intelligence workflows to balance artificial intelligence recommendations with clinician judgment.
- Open Access
- 2 December 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Identifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (Preprint)
BACKGROUND Polypharmacy, the concurrent use of multiple medications, is prevalent among older adults and associated with increased risks for adverse drug events including falls. Deprescribing, the systematic process of discontinuing potentially inappropriate medications, aims to mitigate these risks. However, the practical application of deprescribing criteria in emergency settings remains limited due to time constraints and criteria complexity. OBJECTIVE This study aims to evaluate the performance of a large language model (LLM)–based pipeline in identifying deprescribing opportunities for older emergency department (ED) patients with polypharmacy, using 3 different sets of criteria: Beers, Screening Tool of Older People’s Prescriptions, and Geriatric Emergency Medication Safety Recommendations. The study further evaluates LLM confidence calibration and its ability to improve recommendation performance. METHODS We conducted a retrospective cohort study of older adults presenting to an ED in a large academic medical center in the Northeast United States from January 2022 to March 2022. A random sample of 100 patients (712 total oral medications) was selected for detailed analysis. The LLM pipeline consisted of two steps: (1) filtering high-yield deprescribing criteria based on patients’ medication lists, and (2) applying these criteria using both structured and unstructured patient data to recommend deprescribing. Model performance was assessed by comparing model recommendations to those of trained medical students, with discrepancies adjudicated by board-certified ED physicians. Selective prediction, a method that allows a model to abstain from low-confidence predictions to improve overall reliability, was applied to assess the model’s confidence and decision-making thresholds. RESULTS The LLM was significantly more effective in identifying deprescribing criteria (positive predictive value: 0.83; negative predictive value: 0.93; McNemar test for paired proportions: χ21=5.985; P=.02) relative to medical students, but showed limitations in making specific deprescribing recommendations (positive predictive value=0.47; negative predictive value=0.93). Adjudication revealed that while the model excelled at identifying when there was a deprescribing criterion related to one of the patient’s medications, it often struggled with determining whether that criterion applied to the specific case due to complex inclusion and exclusion criteria (54.5% of errors) and ambiguous clinical contexts (eg, missing information; 39.3% of errors). Selective prediction only marginally improved LLM performance due to poorly calibrated confidence estimates. CONCLUSIONS This study highlights the potential of LLMs to support deprescribing decisions in the ED by effectively filtering relevant criteria. However, challenges remain in applying these criteria to complex clinical scenarios, as the LLM demonstrated poor performance on more intricate decision-making tasks, with its reported confidence often failing to align with its actual success in these cases. The findings underscore the need for clearer deprescribing guidelines, improved LLM calibration for real-world use, and better integration of human–artificial intelligence workflows to balance artificial intelligence recommendations with clinician judgment. CLINICALTRIAL
- 2 December 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessIdentifying Deprescribing Opportunities With Large Language Models in Older Adults: Retrospective Cohort Study (Preprint)
BACKGROUND Polypharmacy, the concurrent use of multiple medications, is prevalent among older adults and associated with increased risks for adverse drug events including falls. Deprescribing, the systematic process of discontinuing potentially inappropriate medications, aims to mitigate these risks. However, the practical application of deprescribing criteria in emergency settings remains limited due to time constraints and criteria complexity. OBJECTIVE This study aims to evaluate the performance of a large language model (LLM)–based pipeline in identifying deprescribing opportunities for older emergency department (ED) patients with polypharmacy, using 3 different sets of criteria: Beers, Screening Tool of Older People’s Prescriptions, and Geriatric Emergency Medication Safety Recommendations. The study further evaluates LLM confidence calibration and its ability to improve recommendation performance. METHODS We conducted a retrospective cohort study of older adults presenting to an ED in a large academic medical center in the Northeast United States from January 2022 to March 2022. A random sample of 100 patients (712 total oral medications) was selected for detailed analysis. The LLM pipeline consisted of two steps: (1) filtering high-yield deprescribing criteria based on patients’ medication lists, and (2) applying these criteria using both structured and unstructured patient data to recommend deprescribing. Model performance was assessed by comparing model recommendations to those of trained medical students, with discrepancies adjudicated by board-certified ED physicians. Selective prediction, a method that allows a model to abstain from low-confidence predictions to improve overall reliability, was applied to assess the model’s confidence and decision-making thresholds. RESULTS The LLM was significantly more effective in identifying deprescribing criteria (positive predictive value: 0.83; negative predictive value: 0.93; McNemar test for paired proportions: χ21=5.985; P=.02) relative to medical students, but showed limitations in making specific deprescribing recommendations (positive predictive value=0.47; negative predictive value=0.93). Adjudication revealed that while the model excelled at identifying when there was a deprescribing criterion related to one of the patient’s medications, it often struggled with determining whether that criterion applied to the specific case due to complex inclusion and exclusion criteria (54.5% of errors) and ambiguous clinical contexts (eg, missing information; 39.3% of errors). Selective prediction only marginally improved LLM performance due to poorly calibrated confidence estimates. CONCLUSIONS This study highlights the potential of LLMs to support deprescribing decisions in the ED by effectively filtering relevant criteria. However, challenges remain in applying these criteria to complex clinical scenarios, as the LLM demonstrated poor performance on more intricate decision-making tasks, with its reported confidence often failing to align with its actual success in these cases. The findings underscore the need for clearer deprescribing guidelines, improved LLM calibration for real-world use, and better integration of human–artificial intelligence workflows to balance artificial intelligence recommendations with clinician judgment. CLINICALTRIAL
- 20 September 2024
- Published by Wiley in Medical Education
- Vol. 59 (3) , 350-351
Dispelling the magic of artificial intelligence in medical education
- 20 September 2024
- Published by Wiley in Medical Education
- Vol. 59 (3) , 350-351
Dispelling the magic of artificial intelligence in medical education