Lea Herschbach
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Lea Herschbach has a total of 16 co-authors. Here are the top co-authors by number of shared publications:
- Anne Herrmann-Werner
Tübingen Institute for Medical Education, University of Tübingen, Tübingen, Germany
11 shared publications - Friederike Holderried
TIME - Tuebingen Institute for Medical Education, University of Tübingen, Tuebingen, Baden-Wurttemberg, Germany
11 shared publications - Teresa Festl-Wietek
TIME - Tübingen Institute for Medical Education, University of Tübingen
11 shared publications - Christian Stegemann-Philipps
Tübingen Institute for Medical Education (TIME), University of Tübingen, Tübingen, Germany
6 shared publications - Jan Griewatz
A Competence Centre for University Teaching in Medicine Baden-Wuerttemberg, Faculty of Medicine , University of Tuebingen , Tuebingen , Germany
6 shared publications - Moritz Mahling
Department of Medical Strategy, Process and Quality Management, University Hospital Tübingen, Tübingen, Germany.
6 shared publications
Latest Publications
- Open Access
- 16 July 2026
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 12, e100467
AI-Simulated Patients for Training Shared Decision-Making: Feasibility Study in Medical Education
Background: Shared decision-making (SDM) is a key element of patient-centered care; however, opportunities for structured and scalable SDM training remain limited in both medical education and clinical practice. Advances in AI have enabled chatbot-based simulations that may support repeated practice and provide automated feedback on SDM-related communication behaviors. Objective: This study aimed to (1) evaluate the agreement between AI-generated and human ratings of SDM performance, (2) compare SDM-related performance and attitudes between medical students and physicians, and (3) explore changes in communicative self-efficacy (CSE) following chatbot-based SDM practice. Methods: We conducted a feasibility study using a pre-post mixed methods design. Medical students and physicians practiced SDM in a 20-minute chat-based interaction with an AI-simulated patient, followed by automated feedback on SDM performance. Prior to the interaction, participants’ attitudes toward SDM (IcanSDM) and their CSE (Self-Efficacy in Patient-Centeredness Questionnaire [SEPCQ-24-GER]) were assessed; CSE was reevaluated after the intervention. SDM performance was measured using the validated Observing Patient Involvement in Decision Making (OPTION-12) scale, with the AI evaluating participants’ performance from the patient’s perspective. These AI-generated ratings were then compared with human ratings to evaluate agreement and rating quality. To further investigate discrepancies between AI and human ratings, a qualitative analysis of selected cases was conducted. Additional measures included demographics, perceived authenticity, and benefits of the interaction. Quantitative analyses included group comparisons, pre-post analyses, and psychometric evaluation of the OPTION-12 scale in an AI-mediated setting. The study was preregistered on the Open Science Framework. Results: A total of 58 participants were included in the analysis, 22 (37.9%) of whom were students, 34 (58.2%) were physicians, and 2 did not declare. Both groups reported similarly positive attitudes toward SDM, but medical students (n=22; mean 28.67, SD 4.32) achieved higher OPTION-12 performance scores than physicians (n=34; mean 22.79, SD 8.14). CSE showed a slight decrease following the intervention. Agreement between AI-generated and human OPTION-12 total scores was high (intraclass correlation coefficient=0.86, df=2,1; P<.001), although AI ratings were systematically higher. The OPTION-12 scale in the AI setting demonstrated good internal consistency (Cronbach α=0.88; mean interitem correlation r=0.43). Participants perceived the AI-simulated interaction as authentic and potentially useful for SDM training. Conclusions: This feasibility study provides preliminary and exploratory insights into the use of AI-simulated patients for SDM training in medical education. The findings contribute to the contextual validation of the OPTION-12 scale in AI-mediated consultations and provide valuable insights into participants’ attitudes toward SDM. However, while findings suggest potential for supporting communication skills development and structured feedback, conclusions are limited by the study design and sample size. Further research with larger samples and controlled designs is needed to establish effectiveness and generalizability.
- 16 July 2026
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 12, e100467
Open AccessAI-Simulated Patients for Training Shared Decision-Making: Feasibility Study in Medical Education
Background: Shared decision-making (SDM) is a key element of patient-centered care; however, opportunities for structured and scalable SDM training remain limited in both medical education and clinical practice. Advances in AI have enabled chatbot-based simulations that may support repeated practice and provide automated feedback on SDM-related communication behaviors. Objective: This study aimed to (1) evaluate the agreement between AI-generated and human ratings of SDM performance, (2) compare SDM-related performance and attitudes between medical students and physicians, and (3) explore changes in communicative self-efficacy (CSE) following chatbot-based SDM practice. Methods: We conducted a feasibility study using a pre-post mixed methods design. Medical students and physicians practiced SDM in a 20-minute chat-based interaction with an AI-simulated patient, followed by automated feedback on SDM performance. Prior to the interaction, participants’ attitudes toward SDM (IcanSDM) and their CSE (Self-Efficacy in Patient-Centeredness Questionnaire [SEPCQ-24-GER]) were assessed; CSE was reevaluated after the intervention. SDM performance was measured using the validated Observing Patient Involvement in Decision Making (OPTION-12) scale, with the AI evaluating participants’ performance from the patient’s perspective. These AI-generated ratings were then compared with human ratings to evaluate agreement and rating quality. To further investigate discrepancies between AI and human ratings, a qualitative analysis of selected cases was conducted. Additional measures included demographics, perceived authenticity, and benefits of the interaction. Quantitative analyses included group comparisons, pre-post analyses, and psychometric evaluation of the OPTION-12 scale in an AI-mediated setting. The study was preregistered on the Open Science Framework. Results: A total of 58 participants were included in the analysis, 22 (37.9%) of whom were students, 34 (58.2%) were physicians, and 2 did not declare. Both groups reported similarly positive attitudes toward SDM, but medical students (n=22; mean 28.67, SD 4.32) achieved higher OPTION-12 performance scores than physicians (n=34; mean 22.79, SD 8.14). CSE showed a slight decrease following the intervention. Agreement between AI-generated and human OPTION-12 total scores was high (intraclass correlation coefficient=0.86, df=2,1; P<.001), although AI ratings were systematically higher. The OPTION-12 scale in the AI setting demonstrated good internal consistency (Cronbach α=0.88; mean interitem correlation r=0.43). Participants perceived the AI-simulated interaction as authentic and potentially useful for SDM training. Conclusions: This feasibility study provides preliminary and exploratory insights into the use of AI-simulated patients for SDM training in medical education. The findings contribute to the contextual validation of the OPTION-12 scale in AI-mediated consultations and provide valuable insights into participants’ attitudes toward SDM. However, while findings suggest potential for supporting communication skills development and structured feedback, conclusions are limited by the study design and sample size. Further research with larger samples and controlled designs is needed to establish effectiveness and generalizability.
- Open Access
- 12 February 2026
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 12, e87102
Using AI to Train Future Clinicians in Depression Assessment: Feasibility Study
Background: Depression is a major global health care challenge, causing significant individual distress but also contributing to a substantial global burden. Timely and accurate diagnosis is crucial. To help future clinicians develop these essential skills, we trained a generative pretrained transformer (GPT)–powered chatbot to simulate patients with varying degrees of depression and suicidality. Objective: This study aims to evaluate the applicability and transferability of our GPT-4-powered chatbot for psychosomatic cases. Specifically, we aim to investigate how accurately the chatbot can simulate patients exhibiting various stages of depression and phases of suicidal ideation, while adhering to a predefined role script and maintaining a sufficient level of authenticity. Additionally, we want to analyze to what level the chatbot is suitable for practicing correctly diagnosing depressive disorders in patients, as well as assessing suicidality stages. Methods: We developed 3 virtual patient role scripts depicting complex, realistic cases of depression and varying degrees of suicidality collaboratively with field experts and aligned with mental health assessment guidelines. These cases were integrated into a GPT-4–powered chatbot for practicing clinical history-taking. A total of 148 medical students, with an average age of 22.71 years and mostly in their sixth semester, interacted individually with one of the randomly assigned virtual patients through chat. Following this, they completed a questionnaire assessing their demographics and user experience. Chats were analyzed descriptively to assess diagnostic accuracy and suicidality assessments, as well as the role script adherence and authenticity of the artificial intelligence (AI). This was done to gain further insight into the chatbot’s behavior and the students’ diagnostic accuracy. Results: In over 90% (725/778) of the answers, the chatbot maintained its assigned role. On average, students correctly identified the severity of depression in 60% (81/135) and the phase of suicidality in 67% (91/135) of the cases. Notably, the majority either failed to address or insufficiently explored the topic of suicidality despite explicit instructions beforehand. Conclusions: This study demonstrates that a GPT-powered chatbot can simulate patients with depression fairly accurately. More than two-thirds of participants perceived the AI-simulated patients with depression as authentic, and nearly 80% (106/135) indicated they would like to use the application for further practice, highlighting its potential as a training tool. While a small proportion of students expressed reservations, and the overall diagnostic accuracy varied depending on the severity of the case, the findings overall support the feasibility and educational value of AI-based role-playing in clinical training. AI-supported virtual patients provide a highly flexible, standardized, and readily available training tool, independent of real-life constraints.
- 12 February 2026
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 12, e87102
Open AccessUsing AI to Train Future Clinicians in Depression Assessment: Feasibility Study
Background: Depression is a major global health care challenge, causing significant individual distress but also contributing to a substantial global burden. Timely and accurate diagnosis is crucial. To help future clinicians develop these essential skills, we trained a generative pretrained transformer (GPT)–powered chatbot to simulate patients with varying degrees of depression and suicidality. Objective: This study aims to evaluate the applicability and transferability of our GPT-4-powered chatbot for psychosomatic cases. Specifically, we aim to investigate how accurately the chatbot can simulate patients exhibiting various stages of depression and phases of suicidal ideation, while adhering to a predefined role script and maintaining a sufficient level of authenticity. Additionally, we want to analyze to what level the chatbot is suitable for practicing correctly diagnosing depressive disorders in patients, as well as assessing suicidality stages. Methods: We developed 3 virtual patient role scripts depicting complex, realistic cases of depression and varying degrees of suicidality collaboratively with field experts and aligned with mental health assessment guidelines. These cases were integrated into a GPT-4–powered chatbot for practicing clinical history-taking. A total of 148 medical students, with an average age of 22.71 years and mostly in their sixth semester, interacted individually with one of the randomly assigned virtual patients through chat. Following this, they completed a questionnaire assessing their demographics and user experience. Chats were analyzed descriptively to assess diagnostic accuracy and suicidality assessments, as well as the role script adherence and authenticity of the artificial intelligence (AI). This was done to gain further insight into the chatbot’s behavior and the students’ diagnostic accuracy. Results: In over 90% (725/778) of the answers, the chatbot maintained its assigned role. On average, students correctly identified the severity of depression in 60% (81/135) and the phase of suicidality in 67% (91/135) of the cases. Notably, the majority either failed to address or insufficiently explored the topic of suicidality despite explicit instructions beforehand. Conclusions: This study demonstrates that a GPT-powered chatbot can simulate patients with depression fairly accurately. More than two-thirds of participants perceived the AI-simulated patients with depression as authentic, and nearly 80% (106/135) indicated they would like to use the application for further practice, highlighting its potential as a training tool. While a small proportion of students expressed reservations, and the overall diagnostic accuracy varied depending on the severity of the case, the findings overall support the feasibility and educational value of AI-based role-playing in clinical training. AI-supported virtual patients provide a highly flexible, standardized, and readily available training tool, independent of real-life constraints.
- Open Access
- 28 November 2025
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 27 (1) , e82818
Evaluation of an AI-Based Chatbot Providing Real-Time Feedback in Communication Training for Mental Health Care Professionals: Proof-of-Concept Observational Study
Background: Effective communication is essential in medical practice, especially in dealing with the increasing number of patients presenting with mental health conditions. Feedback plays a crucial role in improving communication skills but is often difficult to implement. Artificial Intelligence (AI)–based tools offer promising support for practicing and improving communication techniques among physicians. Objective: This proof-of-concept study investigated the capability of an AI-based chatbot that provided physicians with real-time feedback to train communication techniques. Specifically, it addressed three research questions: (1) How accurate is the AI-generated feedback, and how is it perceived by participants? (2) Does the use of an AI-based chatbot and AI-generated feedback lead to an increased frequency in the use of specific communication techniques? (3) Do physicians perceive the training with the AI-based chatbot and AI-generated feedback as beneficial for their daily clinical practice? Methods: The study used a proof-of-concept design using 56 anonymized chat transcripts between physicians and simulated patients. Participating physicians received automated feedback generated by the AI-based chatbot, which was designed to assess and encourage the use of specific communication techniques. Feedback accuracy and user perception were evaluated, and changes in communication behavior were assessed. Finally, participants completed a postintervention questionnaire to evaluate perceived benefits for clinical practice. Results: The AI-generated feedback was found to be very accurate with 85.38% of the real-time feedback being partially or even totally correct. In addition, most of the participants considered the feedback accurate, with 83.64% (n=55) agreeing or totally agreeing. Furthermore, 87.27% (n=55) of the participants agreed or totally agreed to the fact that the feedback was helpful. It also appeared to support the repeated use of recommended communication techniques. Furthermore, most of the participants agreed that practicing with an AI-based chatbot helped them practice and apply new communication techniques in clinical interactions, improved their physician-patient communication, and helped them recognize mental health conditions better in everyday medical practice. Conclusions: The results highlight the potential of AI-supported training to address key communication challenges in medicine, particularly in the context of mental health care. The combination of real-time practice and immediate feedback may foster sustained improvements in physician-patient interactions. Integrating such tools into medical education could offer a valuable complement to traditional training methods. Future research may aim to refine AI models to improve their reliability and investigate the long-term effects and objective measures.
- 28 November 2025
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 27 (1) , e82818
Open AccessEvaluation of an AI-Based Chatbot Providing Real-Time Feedback in Communication Training for Mental Health Care Professionals: Proof-of-Concept Observational Study
Background: Effective communication is essential in medical practice, especially in dealing with the increasing number of patients presenting with mental health conditions. Feedback plays a crucial role in improving communication skills but is often difficult to implement. Artificial Intelligence (AI)–based tools offer promising support for practicing and improving communication techniques among physicians. Objective: This proof-of-concept study investigated the capability of an AI-based chatbot that provided physicians with real-time feedback to train communication techniques. Specifically, it addressed three research questions: (1) How accurate is the AI-generated feedback, and how is it perceived by participants? (2) Does the use of an AI-based chatbot and AI-generated feedback lead to an increased frequency in the use of specific communication techniques? (3) Do physicians perceive the training with the AI-based chatbot and AI-generated feedback as beneficial for their daily clinical practice? Methods: The study used a proof-of-concept design using 56 anonymized chat transcripts between physicians and simulated patients. Participating physicians received automated feedback generated by the AI-based chatbot, which was designed to assess and encourage the use of specific communication techniques. Feedback accuracy and user perception were evaluated, and changes in communication behavior were assessed. Finally, participants completed a postintervention questionnaire to evaluate perceived benefits for clinical practice. Results: The AI-generated feedback was found to be very accurate with 85.38% of the real-time feedback being partially or even totally correct. In addition, most of the participants considered the feedback accurate, with 83.64% (n=55) agreeing or totally agreeing. Furthermore, 87.27% (n=55) of the participants agreed or totally agreed to the fact that the feedback was helpful. It also appeared to support the repeated use of recommended communication techniques. Furthermore, most of the participants agreed that practicing with an AI-based chatbot helped them practice and apply new communication techniques in clinical interactions, improved their physician-patient communication, and helped them recognize mental health conditions better in everyday medical practice. Conclusions: The results highlight the potential of AI-supported training to address key communication challenges in medicine, particularly in the context of mental health care. The combination of real-time practice and immediate feedback may foster sustained improvements in physician-patient interactions. Integrating such tools into medical education could offer a valuable complement to traditional training methods. Future research may aim to refine AI models to improve their reliability and investigate the long-term effects and objective measures.
- Open Access
- 4 November 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Using AI to Train Future Clinicians in Depression Assessment: Feasibility Study (Preprint)
BACKGROUND Depression is a major global health care challenge, causing significant individual distress but also contributing to a substantial global burden. Timely and accurate diagnosis is crucial. To help future clinicians develop these essential skills, we trained a generative pretrained transformer (GPT)–powered chatbot to simulate patients with varying degrees of depression and suicidality. OBJECTIVE This study aims to evaluate the applicability and transferability of our GPT-4-powered chatbot for psychosomatic cases. Specifically, we aim to investigate how accurately the chatbot can simulate patients exhibiting various stages of depression and phases of suicidal ideation, while adhering to a predefined role script and maintaining a sufficient level of authenticity. Additionally, we want to analyze to what level the chatbot is suitable for practicing correctly diagnosing depressive disorders in patients, as well as assessing suicidality stages. METHODS We developed 3 virtual patient role scripts depicting complex, realistic cases of depression and varying degrees of suicidality collaboratively with field experts and aligned with mental health assessment guidelines. These cases were integrated into a GPT-4–powered chatbot for practicing clinical history-taking. A total of 148 medical students, with an average age of 22.71 years and mostly in their sixth semester, interacted individually with one of the randomly assigned virtual patients through chat. Following this, they completed a questionnaire assessing their demographics and user experience. Chats were analyzed descriptively to assess diagnostic accuracy and suicidality assessments, as well as the role script adherence and authenticity of the artificial intelligence (AI). This was done to gain further insight into the chatbot’s behavior and the students’ diagnostic accuracy. RESULTS In over 90% (725/778) of the answers, the chatbot maintained its assigned role. On average, students correctly identified the severity of depression in 60% (81/135) and the phase of suicidality in 67% (91/135) of the cases. Notably, the majority either failed to address or insufficiently explored the topic of suicidality despite explicit instructions beforehand. CONCLUSIONS This study demonstrates that a GPT-powered chatbot can simulate patients with depression fairly accurately. More than two-thirds of participants perceived the AI-simulated patients with depression as authentic, and nearly 80% (106/135) indicated they would like to use the application for further practice, highlighting its potential as a training tool. While a small proportion of students expressed reservations, and the overall diagnostic accuracy varied depending on the severity of the case, the findings overall support the feasibility and educational value of AI-based role-playing in clinical training. AI-supported virtual patients provide a highly flexible, standardized, and readily available training tool, independent of real-life constraints.
- 4 November 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessUsing AI to Train Future Clinicians in Depression Assessment: Feasibility Study (Preprint)
BACKGROUND Depression is a major global health care challenge, causing significant individual distress but also contributing to a substantial global burden. Timely and accurate diagnosis is crucial. To help future clinicians develop these essential skills, we trained a generative pretrained transformer (GPT)–powered chatbot to simulate patients with varying degrees of depression and suicidality. OBJECTIVE This study aims to evaluate the applicability and transferability of our GPT-4-powered chatbot for psychosomatic cases. Specifically, we aim to investigate how accurately the chatbot can simulate patients exhibiting various stages of depression and phases of suicidal ideation, while adhering to a predefined role script and maintaining a sufficient level of authenticity. Additionally, we want to analyze to what level the chatbot is suitable for practicing correctly diagnosing depressive disorders in patients, as well as assessing suicidality stages. METHODS We developed 3 virtual patient role scripts depicting complex, realistic cases of depression and varying degrees of suicidality collaboratively with field experts and aligned with mental health assessment guidelines. These cases were integrated into a GPT-4–powered chatbot for practicing clinical history-taking. A total of 148 medical students, with an average age of 22.71 years and mostly in their sixth semester, interacted individually with one of the randomly assigned virtual patients through chat. Following this, they completed a questionnaire assessing their demographics and user experience. Chats were analyzed descriptively to assess diagnostic accuracy and suicidality assessments, as well as the role script adherence and authenticity of the artificial intelligence (AI). This was done to gain further insight into the chatbot’s behavior and the students’ diagnostic accuracy. RESULTS In over 90% (725/778) of the answers, the chatbot maintained its assigned role. On average, students correctly identified the severity of depression in 60% (81/135) and the phase of suicidality in 67% (91/135) of the cases. Notably, the majority either failed to address or insufficiently explored the topic of suicidality despite explicit instructions beforehand. CONCLUSIONS This study demonstrates that a GPT-powered chatbot can simulate patients with depression fairly accurately. More than two-thirds of participants perceived the AI-simulated patients with depression as authentic, and nearly 80% (106/135) indicated they would like to use the application for further practice, highlighting its potential as a training tool. While a small proportion of students expressed reservations, and the overall diagnostic accuracy varied depending on the severity of the case, the findings overall support the feasibility and educational value of AI-based role-playing in clinical training. AI-supported virtual patients provide a highly flexible, standardized, and readily available training tool, independent of real-life constraints.
- Open Access
- 22 August 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Evaluation of an AI-Based Chatbot Providing Real-Time Feedback in Communication Training for Mental Health Care Professionals: Proof-of-Concept Observational Study (Preprint)
BACKGROUND Effective communication is essential in medical practice, especially in dealing with the increasing number of patients presenting with mental health conditions. Feedback plays a crucial role in improving communication skills but is often difficult to implement. Artificial Intelligence (AI)–based tools offer promising support for practicing and improving communication techniques among physicians. OBJECTIVE This proof-of-concept study investigated the capability of an AI-based chatbot that provided physicians with real-time feedback to train communication techniques. Specifically, it addressed three research questions: (1) How accurate is the AI-generated feedback, and how is it perceived by participants? (2) Does the use of an AI-based chatbot and AI-generated feedback lead to an increased frequency in the use of specific communication techniques? (3) Do physicians perceive the training with the AI-based chatbot and AI-generated feedback as beneficial for their daily clinical practice? METHODS The study used a proof-of-concept design using 56 anonymized chat transcripts between physicians and simulated patients. Participating physicians received automated feedback generated by the AI-based chatbot, which was designed to assess and encourage the use of specific communication techniques. Feedback accuracy and user perception were evaluated, and changes in communication behavior were assessed. Finally, participants completed a postintervention questionnaire to evaluate perceived benefits for clinical practice. RESULTS The AI-generated feedback was found to be very accurate with 85.38% of the real-time feedback being partially or even totally correct. In addition, most of the participants considered the feedback accurate, with 83.64% (n=55) agreeing or totally agreeing. Furthermore, 87.27% (n=55) of the participants agreed or totally agreed to the fact that the feedback was helpful. It also appeared to support the repeated use of recommended communication techniques. Furthermore, most of the participants agreed that practicing with an AI-based chatbot helped them practice and apply new communication techniques in clinical interactions, improved their physician-patient communication, and helped them recognize mental health conditions better in everyday medical practice. CONCLUSIONS The results highlight the potential of AI-supported training to address key communication challenges in medicine, particularly in the context of mental health care. The combination of real-time practice and immediate feedback may foster sustained improvements in physician-patient interactions. Integrating such tools into medical education could offer a valuable complement to traditional training methods. Future research may aim to refine AI models to improve their reliability and investigate the long-term effects and objective measures.
- 22 August 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessEvaluation of an AI-Based Chatbot Providing Real-Time Feedback in Communication Training for Mental Health Care Professionals: Proof-of-Concept Observational Study (Preprint)
BACKGROUND Effective communication is essential in medical practice, especially in dealing with the increasing number of patients presenting with mental health conditions. Feedback plays a crucial role in improving communication skills but is often difficult to implement. Artificial Intelligence (AI)–based tools offer promising support for practicing and improving communication techniques among physicians. OBJECTIVE This proof-of-concept study investigated the capability of an AI-based chatbot that provided physicians with real-time feedback to train communication techniques. Specifically, it addressed three research questions: (1) How accurate is the AI-generated feedback, and how is it perceived by participants? (2) Does the use of an AI-based chatbot and AI-generated feedback lead to an increased frequency in the use of specific communication techniques? (3) Do physicians perceive the training with the AI-based chatbot and AI-generated feedback as beneficial for their daily clinical practice? METHODS The study used a proof-of-concept design using 56 anonymized chat transcripts between physicians and simulated patients. Participating physicians received automated feedback generated by the AI-based chatbot, which was designed to assess and encourage the use of specific communication techniques. Feedback accuracy and user perception were evaluated, and changes in communication behavior were assessed. Finally, participants completed a postintervention questionnaire to evaluate perceived benefits for clinical practice. RESULTS The AI-generated feedback was found to be very accurate with 85.38% of the real-time feedback being partially or even totally correct. In addition, most of the participants considered the feedback accurate, with 83.64% (n=55) agreeing or totally agreeing. Furthermore, 87.27% (n=55) of the participants agreed or totally agreed to the fact that the feedback was helpful. It also appeared to support the repeated use of recommended communication techniques. Furthermore, most of the participants agreed that practicing with an AI-based chatbot helped them practice and apply new communication techniques in clinical interactions, improved their physician-patient communication, and helped them recognize mental health conditions better in everyday medical practice. CONCLUSIONS The results highlight the potential of AI-supported training to address key communication challenges in medicine, particularly in the context of mental health care. The combination of real-time practice and immediate feedback may foster sustained improvements in physician-patient interactions. Integrating such tools into medical education could offer a valuable complement to traditional training methods. Future research may aim to refine AI models to improve their reliability and investigate the long-term effects and objective measures.
- Open Access
- 16 April 2024
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 26 (1) , e57778
Authors’ Reply: “Evaluating GPT-4’s Cognitive Functions Through the Bloom Taxonomy: Insights and Clarifications”
- 16 April 2024
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 26 (1) , e57778
Open AccessAuthors’ Reply: “Evaluating GPT-4’s Cognitive Functions Through the Bloom Taxonomy: Insights and Clarifications”
- Open Access
- 26 February 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Authors’ Reply: “Evaluating GPT-4’s Cognitive Functions Through the Bloom Taxonomy: Insights and Clarifications” (Preprint)
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author).
- 26 February 2024
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessAuthors’ Reply: “Evaluating GPT-4’s Cognitive Functions Through the Bloom Taxonomy: Insights and Clarifications” (Preprint)
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author).
- Top Cited PapersOpen Access
- 23 January 2024
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 26 (1) , e52113
Assessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study
Background: Large language models such as GPT-4 (Generative Pre-trained Transformer 4) are being increasingly used in medicine and medical education. However, these models are prone to “hallucinations” (ie, outputs that seem convincing while being factually incorrect). It is currently unknown how these errors by large language models relate to the different cognitive levels defined in Bloom’s taxonomy. Objective: This study aims to explore how GPT-4 performs in terms of Bloom’s taxonomy using psychosomatic medicine exam questions. Methods: We used a large data set of psychosomatic medicine multiple-choice questions (N=307) with real-world results derived from medical school exams. GPT-4 answered the multiple-choice questions using 2 distinct prompt versions: detailed and short. The answers were analyzed using a quantitative approach and a qualitative approach. Focusing on incorrectly answered questions, we categorized reasoning errors according to the hierarchical framework of Bloom’s taxonomy. Results: GPT-4’s performance in answering exam questions yielded a high success rate: 93% (284/307) for the detailed prompt and 91% (278/307) for the short prompt. Questions answered correctly by GPT-4 had a statistically significant higher difficulty than questions answered incorrectly (P=.002 for the detailed prompt and P<.001 for the short prompt). Independent of the prompt, GPT-4’s lowest exam performance was 78.9% (15/19), thereby always surpassing the “pass” threshold. Our qualitative analysis of incorrect answers, based on Bloom’s taxonomy, showed that errors were primarily in the “remember” (29/68) and “understand” (23/68) cognitive levels; specific issues arose in recalling details, understanding conceptual relationships, and adhering to standardized guidelines. Conclusions: GPT-4 demonstrated a remarkable success rate when confronted with psychosomatic medicine multiple-choice exam questions, aligning with previous findings. When evaluated through Bloom’s taxonomy, our data revealed that GPT-4 occasionally ignored specific facts (remember), provided illogical reasoning (understand), or failed to apply concepts to a new situation (apply). These errors, which were confidently presented, could be attributed to inherent model biases and the tendency to generate outputs that maximize likelihood.
- 23 January 2024
- Published by JMIR Publications Inc. in Journal of Medical Internet Research
- Vol. 26 (1) , e52113
Top Cited PapersOpen AccessAssessing ChatGPT’s Mastery of Bloom’s Taxonomy Using Psychosomatic Medicine Exam Questions: Mixed-Methods Study
Background: Large language models such as GPT-4 (Generative Pre-trained Transformer 4) are being increasingly used in medicine and medical education. However, these models are prone to “hallucinations” (ie, outputs that seem convincing while being factually incorrect). It is currently unknown how these errors by large language models relate to the different cognitive levels defined in Bloom’s taxonomy. Objective: This study aims to explore how GPT-4 performs in terms of Bloom’s taxonomy using psychosomatic medicine exam questions. Methods: We used a large data set of psychosomatic medicine multiple-choice questions (N=307) with real-world results derived from medical school exams. GPT-4 answered the multiple-choice questions using 2 distinct prompt versions: detailed and short. The answers were analyzed using a quantitative approach and a qualitative approach. Focusing on incorrectly answered questions, we categorized reasoning errors according to the hierarchical framework of Bloom’s taxonomy. Results: GPT-4’s performance in answering exam questions yielded a high success rate: 93% (284/307) for the detailed prompt and 91% (278/307) for the short prompt. Questions answered correctly by GPT-4 had a statistically significant higher difficulty than questions answered incorrectly (P=.002 for the detailed prompt and P<.001 for the short prompt). Independent of the prompt, GPT-4’s lowest exam performance was 78.9% (15/19), thereby always surpassing the “pass” threshold. Our qualitative analysis of incorrect answers, based on Bloom’s taxonomy, showed that errors were primarily in the “remember” (29/68) and “understand” (23/68) cognitive levels; specific issues arose in recalling details, understanding conceptual relationships, and adhering to standardized guidelines. Conclusions: GPT-4 demonstrated a remarkable success rate when confronted with psychosomatic medicine multiple-choice exam questions, aligning with previous findings. When evaluated through Bloom’s taxonomy, our data revealed that GPT-4 occasionally ignored specific facts (remember), provided illogical reasoning (understand), or failed to apply concepts to a new situation (apply). These errors, which were confidently presented, could be attributed to inherent model biases and the tendency to generate outputs that maximize likelihood.
- Top Cited PapersOpen Access
- 16 January 2024
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 10 (1) , e53961
A Generative Pretrained Transformer (GPT)–Powered Chatbot as a Simulated Patient to Practice History Taking: Prospective, Mixed Methods Study
Background: Communication is a core competency of medical professionals and of utmost importance for patient safety. Although medical curricula emphasize communication training, traditional formats, such as real or simulated patient interactions, can present psychological stress and are limited in repetition. The recent emergence of large language models (LLMs), such as generative pretrained transformer (GPT), offers an opportunity to overcome these restrictions Objective: The aim of this study was to explore the feasibility of a GPT-driven chatbot to practice history taking, one of the core competencies of communication. Methods: We developed an interactive chatbot interface using GPT-3.5 and a specific prompt including a chatbot-optimized illness script and a behavioral component. Following a mixed methods approach, we invited medical students to voluntarily practice history taking. To determine whether GPT provides suitable answers as a simulated patient, the conversations were recorded and analyzed using quantitative and qualitative approaches. We analyzed the extent to which the questions and answers aligned with the provided script, as well as the medical plausibility of the answers. Finally, the students filled out the Chatbot Usability Questionnaire (CUQ). Results: A total of 28 students practiced with our chatbot (mean age 23.4, SD 2.9 years). We recorded a total of 826 question-answer pairs (QAPs), with a median of 27.5 QAPs per conversation and 94.7% (n=782) pertaining to history taking. When questions were explicitly covered by the script (n=502, 60.3%), the GPT-provided answers were mostly based on explicit script information (n=471, 94.4%). For questions not covered by the script (n=195, 23.4%), the GPT answers used 56.4% (n=110) fictitious information. Regarding plausibility, 842 (97.9%) of 860 QAPs were rated as plausible. Of the 14 (2.1%) implausible answers, GPT provided answers rated as socially desirable, leaving role identity, ignoring script information, illogical reasoning, and calculation error. Despite these results, the CUQ revealed an overall positive user experience (77/100 points). Conclusions: Our data showed that LLMs, such as GPT, can provide a simulated patient experience and yield a good user experience and a majority of plausible answers. Our analysis revealed that GPT-provided answers use either explicit script information or are based on available information, which can be understood as abductive reasoning. Although rare, the GPT-based chatbot provides implausible information in some instances, with the major tendency being socially desirable instead of medically plausible information.
- 16 January 2024
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 10 (1) , e53961
Top Cited PapersOpen AccessA Generative Pretrained Transformer (GPT)–Powered Chatbot as a Simulated Patient to Practice History Taking: Prospective, Mixed Methods Study
Background: Communication is a core competency of medical professionals and of utmost importance for patient safety. Although medical curricula emphasize communication training, traditional formats, such as real or simulated patient interactions, can present psychological stress and are limited in repetition. The recent emergence of large language models (LLMs), such as generative pretrained transformer (GPT), offers an opportunity to overcome these restrictions Objective: The aim of this study was to explore the feasibility of a GPT-driven chatbot to practice history taking, one of the core competencies of communication. Methods: We developed an interactive chatbot interface using GPT-3.5 and a specific prompt including a chatbot-optimized illness script and a behavioral component. Following a mixed methods approach, we invited medical students to voluntarily practice history taking. To determine whether GPT provides suitable answers as a simulated patient, the conversations were recorded and analyzed using quantitative and qualitative approaches. We analyzed the extent to which the questions and answers aligned with the provided script, as well as the medical plausibility of the answers. Finally, the students filled out the Chatbot Usability Questionnaire (CUQ). Results: A total of 28 students practiced with our chatbot (mean age 23.4, SD 2.9 years). We recorded a total of 826 question-answer pairs (QAPs), with a median of 27.5 QAPs per conversation and 94.7% (n=782) pertaining to history taking. When questions were explicitly covered by the script (n=502, 60.3%), the GPT-provided answers were mostly based on explicit script information (n=471, 94.4%). For questions not covered by the script (n=195, 23.4%), the GPT answers used 56.4% (n=110) fictitious information. Regarding plausibility, 842 (97.9%) of 860 QAPs were rated as plausible. Of the 14 (2.1%) implausible answers, GPT provided answers rated as socially desirable, leaving role identity, ignoring script information, illogical reasoning, and calculation error. Despite these results, the CUQ revealed an overall positive user experience (77/100 points). Conclusions: Our data showed that LLMs, such as GPT, can provide a simulated patient experience and yield a good user experience and a majority of plausible answers. Our analysis revealed that GPT-provided answers use either explicit script information or are based on available information, which can be understood as abductive reasoning. Although rare, the GPT-based chatbot provides implausible information in some instances, with the major tendency being socially desirable instead of medically plausible information.
- Open Access
- 25 October 2023
- Published by JMIR Publications Inc. in JMIR Preprints
A Generative Pretrained Transformer (GPT)–Powered Chatbot as a Simulated Patient to Practice History Taking: Prospective, Mixed Methods Study (Preprint)
BACKGROUND Communication is a core competency of medical professionals and of utmost importance for patient safety. Although medical curricula emphasize communication training, traditional formats, such as real or simulated patient interactions, can present psychological stress and are limited in repetition. The recent emergence of large language models (LLMs), such as generative pretrained transformer (GPT), offers an opportunity to overcome these restrictions OBJECTIVE The aim of this study was to explore the feasibility of a GPT-driven chatbot to practice history taking, one of the core competencies of communication. METHODS We developed an interactive chatbot interface using GPT-3.5 and a specific prompt including a chatbot-optimized illness script and a behavioral component. Following a mixed methods approach, we invited medical students to voluntarily practice history taking. To determine whether GPT provides suitable answers as a simulated patient, the conversations were recorded and analyzed using quantitative and qualitative approaches. We analyzed the extent to which the questions and answers aligned with the provided script, as well as the medical plausibility of the answers. Finally, the students filled out the Chatbot Usability Questionnaire (CUQ). RESULTS A total of 28 students practiced with our chatbot (mean age 23.4, SD 2.9 years). We recorded a total of 826 question-answer pairs (QAPs), with a median of 27.5 QAPs per conversation and 94.7% (n=782) pertaining to history taking. When questions were explicitly covered by the script (n=502, 60.3%), the GPT-provided answers were mostly based on explicit script information (n=471, 94.4%). For questions not covered by the script (n=195, 23.4%), the GPT answers used 56.4% (n=110) fictitious information. Regarding plausibility, 842 (97.9%) of 860 QAPs were rated as plausible. Of the 14 (2.1%) implausible answers, GPT provided answers rated as socially desirable, leaving role identity, ignoring script information, illogical reasoning, and calculation error. Despite these results, the CUQ revealed an overall positive user experience (77/100 points). CONCLUSIONS Our data showed that LLMs, such as GPT, can provide a simulated patient experience and yield a good user experience and a majority of plausible answers. Our analysis revealed that GPT-provided answers use either explicit script information or are based on available information, which can be understood as abductive reasoning. Although rare, the GPT-based chatbot provides implausible information in some instances, with the major tendency being socially desirable instead of medically plausible information.
- 25 October 2023
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessA Generative Pretrained Transformer (GPT)–Powered Chatbot as a Simulated Patient to Practice History Taking: Prospective, Mixed Methods Study (Preprint)
BACKGROUND Communication is a core competency of medical professionals and of utmost importance for patient safety. Although medical curricula emphasize communication training, traditional formats, such as real or simulated patient interactions, can present psychological stress and are limited in repetition. The recent emergence of large language models (LLMs), such as generative pretrained transformer (GPT), offers an opportunity to overcome these restrictions OBJECTIVE The aim of this study was to explore the feasibility of a GPT-driven chatbot to practice history taking, one of the core competencies of communication. METHODS We developed an interactive chatbot interface using GPT-3.5 and a specific prompt including a chatbot-optimized illness script and a behavioral component. Following a mixed methods approach, we invited medical students to voluntarily practice history taking. To determine whether GPT provides suitable answers as a simulated patient, the conversations were recorded and analyzed using quantitative and qualitative approaches. We analyzed the extent to which the questions and answers aligned with the provided script, as well as the medical plausibility of the answers. Finally, the students filled out the Chatbot Usability Questionnaire (CUQ). RESULTS A total of 28 students practiced with our chatbot (mean age 23.4, SD 2.9 years). We recorded a total of 826 question-answer pairs (QAPs), with a median of 27.5 QAPs per conversation and 94.7% (n=782) pertaining to history taking. When questions were explicitly covered by the script (n=502, 60.3%), the GPT-provided answers were mostly based on explicit script information (n=471, 94.4%). For questions not covered by the script (n=195, 23.4%), the GPT answers used 56.4% (n=110) fictitious information. Regarding plausibility, 842 (97.9%) of 860 QAPs were rated as plausible. Of the 14 (2.1%) implausible answers, GPT provided answers rated as socially desirable, leaving role identity, ignoring script information, illogical reasoning, and calculation error. Despite these results, the CUQ revealed an overall positive user experience (77/100 points). CONCLUSIONS Our data showed that LLMs, such as GPT, can provide a simulated patient experience and yield a good user experience and a majority of plausible answers. Our analysis revealed that GPT-provided answers use either explicit script information or are based on available information, which can be understood as abductive reasoning. Although rare, the GPT-based chatbot provides implausible information in some instances, with the major tendency being socially desirable instead of medically plausible information.