Yuji Yamada
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Research Interests
Yuji Yamada has a total of 80 co-authors. Here are the top co-authors by number of shared publications:
- Toshio Naito
Department of General Medicine, Juntendo University Faculty of Medicine, Tokyo, Japan
5 shared publications - Hiromizu Takahashi
Department of General Medicine, Faculty of Medicine, Juntendo University, 3-1-3 Hongo, Bunkyo Tokyo, 1130033 Japan, Tokyo, Japan, 81 3-3813-3111
4 shared publications - Kiyoshi Shikino
Department of Community-Oriented Medical Education, Chiba University Graduate School of Medicine School of Medicine, Chiba, Japan
4 shared publications - Takeshi Kondo
Nagoya University Graduate School of Medicine
4 shared publications - Akira Komori
Department of Emergency and Critical Care Medicine, Tsukuba Memorial Hospital, Tsukuba, Japan
2 shared publications - Alexandre Rodrigues Marra
Hospital Israelita Albert Einstein
2 shared publications
Latest Publications
- Open Access
- 23 July 2026
- Published by Wiley in Geriatrics & Gerontology International
- Vol. 26 (7)
Advancing Geriatric Care in Japan: Integrating the 5Ms Framework With Comprehensive Geriatric Assessment for Enhanced Clinical Practice in a Super‐Aged Society
Background: Japan faces unprecedented demographic challenges as the world's most rapidly aging country, with 29.4% of the population aged ≥ 65 (2025) and projections estimating 38% by 2065. While Comprehensive Geriatric Assessment (CGA) is the gold standard for geriatric care, its implementation in Japan remains limited to geriatric specialists, creating a significant gap in delivering optimal care to Japan's growing older adult population. Objective: To propose an integrated model that combines the 5Ms framework (Mind, Mobility, Medications, Multicomplexity, Matters Most) with CGA to enhance accessibility, practicality, and continuity of geriatric care in Japanese healthcare systems. Methods: This position statement synthesizes evidence on CGA effectiveness and the educational and clinical utility of the 5Ms, evaluates cultural considerations in Japan, and proposes a three‐tiered implementation model. Results: The integration of CGA and the 5Ms framework creates a tiered assessment continuum that adapts to clinical contexts, facilitates team‐based care, and improves quality and continuity across settings. The model supports education, quality improvement, and culturally sensitive care planning. Conclusions: The integration of the 5Ms and CGA offers a practical framework to formalize, connect, and strengthen geriatric care already being delivered across Japanese healthcare settings. The Japan Geriatrics Society supports this model as a guiding structure for phased implementation, workforce development, and ongoing evaluation to enhance the delivery of person‐centered geriatric care in Japan.
- 23 July 2026
- Published by Wiley in Geriatrics & Gerontology International
- Vol. 26 (7)
Open AccessAdvancing Geriatric Care in Japan: Integrating the 5Ms Framework With Comprehensive Geriatric Assessment for Enhanced Clinical Practice in a Super‐Aged Society
Background: Japan faces unprecedented demographic challenges as the world's most rapidly aging country, with 29.4% of the population aged ≥ 65 (2025) and projections estimating 38% by 2065. While Comprehensive Geriatric Assessment (CGA) is the gold standard for geriatric care, its implementation in Japan remains limited to geriatric specialists, creating a significant gap in delivering optimal care to Japan's growing older adult population. Objective: To propose an integrated model that combines the 5Ms framework (Mind, Mobility, Medications, Multicomplexity, Matters Most) with CGA to enhance accessibility, practicality, and continuity of geriatric care in Japanese healthcare systems. Methods: This position statement synthesizes evidence on CGA effectiveness and the educational and clinical utility of the 5Ms, evaluates cultural considerations in Japan, and proposes a three‐tiered implementation model. Results: The integration of CGA and the 5Ms framework creates a tiered assessment continuum that adapts to clinical contexts, facilitates team‐based care, and improves quality and continuity across settings. The model supports education, quality improvement, and culturally sensitive care planning. Conclusions: The integration of the 5Ms and CGA offers a practical framework to formalize, connect, and strengthen geriatric care already being delivered across Japanese healthcare settings. The Japan Geriatrics Society supports this model as a guiding structure for phased implementation, workforce development, and ongoing evaluation to enhance the delivery of person‐centered geriatric care in Japan.
- 19 March 2026
- Published by Elsevier in The American Journal of Medicine
- Vol. 139 (8) , 972-978.e5
Association between pet ownership and incident dementia: A systematic review
- 19 March 2026
- Published by Elsevier in The American Journal of Medicine
- Vol. 139 (8) , 972-978.e5
Association between pet ownership and incident dementia: A systematic review
- Top Cited PapersOpen Access
- 17 February 2026
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 12, e81673
AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI–Simulated Patient System: Cross-Sectional Validation Study
Background: Generative artificial intelligence (AI) is increasingly used in medical education, including AI-based virtual patients to improve interview skills. However, how much AI-based assessment (ABA) differs from human-based assessment (HBA) remains unclear. Objective: This study aimed to compare the quality of clinical interview assessments generated via an ABA (GPT-o1 Pro [ABA-o1] and GPT-5 Pro [ABA-5]) with those generated via an HBA conducted by clinical instructors in an AI-based virtual patient setting. We also examined whether AI reduced evaluation time and assessed agreement across participants with different levels of clinical experience. Methods: A standardized case of leg weakness was implemented in an AI-based virtual patient. Seven participants (2 medical students, 3 residents, and 2 attending physicians) each conducted an interview with the AI patient, and transcripts were scored using the 25-item Master Interview Rating Scale (0‐125). Three evaluation strategies were compared. First, GPT-o1 Pro and GPT-5 Pro scored each transcript 5 times with different random seeds to test case specificity. Processing time was logged automatically. Second, 5 blinded clinical instructors independently rated each transcript once using the same rubric. Third, reliability metrics were applied. For AI, intraclass correlation coefficients (ICCs) quantified repeatability. For humans, the ICC(2,1) was calculated. Agreement was quantified using the Pearson r , Lin concordance correlation coefficient, Bland-Altman limits of agreement, Cronbach α, and ICC. Time efficiency was expressed as mean minutes per transcript and relative percentage reduction. Results: Mean interview scores were similar across methods (ABA-o1: mean 52.1, SD 6.9; ABA-5: mean 53.2, SD 6.8; HBA: mean 53.7, SD 6.8). Agreement between ABA and HBA was strong ( r =0.90; concordance correlation coefficient=0.88) with minimal bias (ABA-o1: mean 0.4, SD 2.7; ABA-5: mean 1.5, SD 5.2; limits of agreement: –4.9 to 5.7 for ABA-o1 and –8.6 to 11.7 for ABA-5). The Cronbach α was 0.81 (ABA-o1), 0.86 (ABA-5), and 0.80 (HBA); the ICC(3,1) was 0.77 (ABA-o1) and 0.82 (ABA-5); and the ICC(2,1) was 0.38 (HBA). The coefficient of variation for ABA was approximately half that of HBA (6.6% vs 13.9%). Processing time for 5 runs was 4 minutes, 19 seconds for ABA-o1 and 3 minutes, 20 seconds for ABA-5 vs 10 minutes, 16 seconds for physicians, corresponding to 58% and 67.6% reductions, respectively. Conclusions: ABA-o1 and ABA-5 produced scores closely matching HBA while demonstrating superior consistency and reliability. In the setting of virtual interview transcripts, these findings suggest that ABA may serve as a valid, rapid, and scalable alternative to HBA, reducing per-assessment time by over half. Applied strategically, AI-based scoring could enable timely feedback, improve efficiency, and reduce faculty workload. Further research is needed to confirm generalizability across broader settings.
- 17 February 2026
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 12, e81673
Top Cited PapersOpen AccessAI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI–Simulated Patient System: Cross-Sectional Validation Study
Background: Generative artificial intelligence (AI) is increasingly used in medical education, including AI-based virtual patients to improve interview skills. However, how much AI-based assessment (ABA) differs from human-based assessment (HBA) remains unclear. Objective: This study aimed to compare the quality of clinical interview assessments generated via an ABA (GPT-o1 Pro [ABA-o1] and GPT-5 Pro [ABA-5]) with those generated via an HBA conducted by clinical instructors in an AI-based virtual patient setting. We also examined whether AI reduced evaluation time and assessed agreement across participants with different levels of clinical experience. Methods: A standardized case of leg weakness was implemented in an AI-based virtual patient. Seven participants (2 medical students, 3 residents, and 2 attending physicians) each conducted an interview with the AI patient, and transcripts were scored using the 25-item Master Interview Rating Scale (0‐125). Three evaluation strategies were compared. First, GPT-o1 Pro and GPT-5 Pro scored each transcript 5 times with different random seeds to test case specificity. Processing time was logged automatically. Second, 5 blinded clinical instructors independently rated each transcript once using the same rubric. Third, reliability metrics were applied. For AI, intraclass correlation coefficients (ICCs) quantified repeatability. For humans, the ICC(2,1) was calculated. Agreement was quantified using the Pearson r , Lin concordance correlation coefficient, Bland-Altman limits of agreement, Cronbach α, and ICC. Time efficiency was expressed as mean minutes per transcript and relative percentage reduction. Results: Mean interview scores were similar across methods (ABA-o1: mean 52.1, SD 6.9; ABA-5: mean 53.2, SD 6.8; HBA: mean 53.7, SD 6.8). Agreement between ABA and HBA was strong ( r =0.90; concordance correlation coefficient=0.88) with minimal bias (ABA-o1: mean 0.4, SD 2.7; ABA-5: mean 1.5, SD 5.2; limits of agreement: –4.9 to 5.7 for ABA-o1 and –8.6 to 11.7 for ABA-5). The Cronbach α was 0.81 (ABA-o1), 0.86 (ABA-5), and 0.80 (HBA); the ICC(3,1) was 0.77 (ABA-o1) and 0.82 (ABA-5); and the ICC(2,1) was 0.38 (HBA). The coefficient of variation for ABA was approximately half that of HBA (6.6% vs 13.9%). Processing time for 5 runs was 4 minutes, 19 seconds for ABA-o1 and 3 minutes, 20 seconds for ABA-5 vs 10 minutes, 16 seconds for physicians, corresponding to 58% and 67.6% reductions, respectively. Conclusions: ABA-o1 and ABA-5 produced scores closely matching HBA while demonstrating superior consistency and reliability. In the setting of virtual interview transcripts, these findings suggest that ABA may serve as a valid, rapid, and scalable alternative to HBA, reducing per-assessment time by over half. Applied strategically, AI-based scoring could enable timely feedback, improve efficiency, and reduce faculty workload. Further research is needed to confirm generalizability across broader settings.
- Open Access
- 21 August 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Comparing AI- and Human-Based Assessments of Medical Interview Transcripts Using a Generative AI Simulated Patient System. (Preprint)
BACKGROUND Generative AI is increasingly used in medical education, including the use of AI-based virtual patients to improve interview skills. However, it remains unclear how much AI-based assessment (ABA) differs from those of Human-based assessment (HBA). OBJECTIVE This study aimed to compare the quality of clinical interview assessments generated by an ABA using a virtual patient with those provided by a HBA conducted by clinical instructors. Additionally, it evaluated whether the use of AI could lead to a measurable reduction in evaluation time, and examined the level of agreement across participants with differing levels of clinical experience. METHODS A standardized leg-weakness case was implemented in an AI based virtual patient. Seven participants—two medical students, three resident physicians, and two attending physicians—each conducted an interview, and transcripts were scored with the Master Interview Rating Scale (MIRS; 25 items, 0–5 scale; total 0–125). Two evaluation strategies were compared. (1) ChatGPT o1-Pro scored each transcript five times with different random seeds to assess case specificity; total runtime for the five scores was automatically logged. (2) Five blinded clinical instructors , after a preparatory webinar reviewing the rubric and practicing on sample transcripts, each rated every transcript once and recorded clock time per rating. Because the five AI outputs are replicates of the same algorithm, intraclass correlation coefficients (ICC) were used to quantify repeatability rather than inter rater reliability. For human raters, we calculated ICC (2,1). Mean scores from both methods were compared, and agreement was quantified with Pearson’s r, Lin’s concordance correlation coefficient (ρc), Bland–Altman limits of agreement (LoA), internal consistency (Cronbach’s α), and ICC. Time efficiency was expressed as mean minutes per transcript and the relative percentage reduction achieved by AI scoring. RESULTS Mean interview scores were similar for ABA and HBA (52.1 ± 6.9 vs 53.7 ± 6.8). Agreement was strong (r = 0.92; ρc = 0.92) with minimal bias (+0.4 points; LoA −4.9 to +5.7). ABA showed higher internal consistency (α = 0.936 vs 0.863) and greater inter rater reliability (ICC = 0.77 vs 0.38). The coefficient of variation for ABA scores was roughly half that of HBA scores (6.6 % vs 13.9 %). In addition, ChatGPT completed each five run analysis in 4.3 ± 1.7 minutes compared with 10.3 ± 3.3 minutes for physicians, representing a 58 % reduction in assessment time. CONCLUSIONS ABA scores that closely matched HBA scores while demonstrating superior consistency and reliability. In the setting of virtual clinical interview transcripts, these preliminary findings suggest that ABA shows potential as a valid, rapid, and scalable alternative to HBA. When applied strategically, it could potentially furnish timely formative feedback, quantify efficiency gains, and reduce faculty workload without compromising assessment quality. Further research is needed to determine whether this can be achieved without compromising assessment quality.
- 21 August 2025
- Published by JMIR Publications Inc. in JMIR Preprints
Open AccessComparing AI- and Human-Based Assessments of Medical Interview Transcripts Using a Generative AI Simulated Patient System. (Preprint)
BACKGROUND Generative AI is increasingly used in medical education, including the use of AI-based virtual patients to improve interview skills. However, it remains unclear how much AI-based assessment (ABA) differs from those of Human-based assessment (HBA). OBJECTIVE This study aimed to compare the quality of clinical interview assessments generated by an ABA using a virtual patient with those provided by a HBA conducted by clinical instructors. Additionally, it evaluated whether the use of AI could lead to a measurable reduction in evaluation time, and examined the level of agreement across participants with differing levels of clinical experience. METHODS A standardized leg-weakness case was implemented in an AI based virtual patient. Seven participants—two medical students, three resident physicians, and two attending physicians—each conducted an interview, and transcripts were scored with the Master Interview Rating Scale (MIRS; 25 items, 0–5 scale; total 0–125). Two evaluation strategies were compared. (1) ChatGPT o1-Pro scored each transcript five times with different random seeds to assess case specificity; total runtime for the five scores was automatically logged. (2) Five blinded clinical instructors , after a preparatory webinar reviewing the rubric and practicing on sample transcripts, each rated every transcript once and recorded clock time per rating. Because the five AI outputs are replicates of the same algorithm, intraclass correlation coefficients (ICC) were used to quantify repeatability rather than inter rater reliability. For human raters, we calculated ICC (2,1). Mean scores from both methods were compared, and agreement was quantified with Pearson’s r, Lin’s concordance correlation coefficient (ρc), Bland–Altman limits of agreement (LoA), internal consistency (Cronbach’s α), and ICC. Time efficiency was expressed as mean minutes per transcript and the relative percentage reduction achieved by AI scoring. RESULTS Mean interview scores were similar for ABA and HBA (52.1 ± 6.9 vs 53.7 ± 6.8). Agreement was strong (r = 0.92; ρc = 0.92) with minimal bias (+0.4 points; LoA −4.9 to +5.7). ABA showed higher internal consistency (α = 0.936 vs 0.863) and greater inter rater reliability (ICC = 0.77 vs 0.38). The coefficient of variation for ABA scores was roughly half that of HBA scores (6.6 % vs 13.9 %). In addition, ChatGPT completed each five run analysis in 4.3 ± 1.7 minutes compared with 10.3 ± 3.3 minutes for physicians, representing a 58 % reduction in assessment time. CONCLUSIONS ABA scores that closely matched HBA scores while demonstrating superior consistency and reliability. In the setting of virtual clinical interview transcripts, these preliminary findings suggest that ABA shows potential as a valid, rapid, and scalable alternative to HBA. When applied strategically, it could potentially furnish timely formative feedback, quantify efficiency gains, and reduce faculty workload without compromising assessment quality. Further research is needed to determine whether this can be achieved without compromising assessment quality.
- Open Access
- 7 August 2025
- Published by Cambridge University Press (CUP) in Antimicrobial Stewardship & Healthcare Epidemiology
- Vol. 5 (1) , e178
Fournier gangrene is associated with increased length of stay and higher healthcare costs compared to non-perineal necrotizing soft tissue infections: a retrospective analysis of the National Inpatient Sample (2016–2020)
Background: Fournier gangrene (FG) is a necrotizing soft tissue infection (NSTI) of the perineum. Recent retrospective studies from quaternary centers suggest improved outcomes and a potentially less aggressive clinical course for FG than non-perineal NSTIs. However, comprehensive nationwide data remain limited. Methods: This retrospective cohort study analyzed the National Inpatient Sample (2016–2020) to compare outcomes between FG and non-perineal NSTIs. Adult patients undergoing surgical debridement with a diagnosis of FG or NSTI were identified using ICD-10 codes. Outcomes included in-hospital mortality, length of stay (LOS), hospital costs, and home discharge rates. Multivariable regression analyses adjusted for patient demographics, comorbidities, and hospital characteristics. Results: A total of 5,007 FG and 24,782 non-perineal NSTI patients were identified. Crude in-hospital mortality rates were 5.8% for FG and 5.4% for non-perineal NSTIs, with stable trends observed over five years. After adjustment, no significant difference in mortality was observed (adjusted odds ratio [aOR]: 1.04; 95% CI: 0.90–1.20). However, FG was associated with longer LOS (adjusted mean difference: 1.99 days; 95% CI: 1.53–2.46) and higher hospital costs ($37,809 higher; 95% CI: $29,540–$46,077). Home discharge rates were similar between groups (aOR: 0.97; 95% CI: 0.89–1.05). Discussion: Despite similar mortality rates, FG hospitalizations were associated with increased LOS and higher costs compared to non-perineal NSTIs. These findings may suggest potential nationwide disparities in FG care quality, particularly outside specialized referral centers. Further research is needed to understand if standardized care pathways tailored to FG may optimize management and reduce resource utilization.
- 7 August 2025
- Published by Cambridge University Press (CUP) in Antimicrobial Stewardship & Healthcare Epidemiology
- Vol. 5 (1) , e178
Open AccessFournier gangrene is associated with increased length of stay and higher healthcare costs compared to non-perineal necrotizing soft tissue infections: a retrospective analysis of the National Inpatient Sample (2016–2020)
Background: Fournier gangrene (FG) is a necrotizing soft tissue infection (NSTI) of the perineum. Recent retrospective studies from quaternary centers suggest improved outcomes and a potentially less aggressive clinical course for FG than non-perineal NSTIs. However, comprehensive nationwide data remain limited. Methods: This retrospective cohort study analyzed the National Inpatient Sample (2016–2020) to compare outcomes between FG and non-perineal NSTIs. Adult patients undergoing surgical debridement with a diagnosis of FG or NSTI were identified using ICD-10 codes. Outcomes included in-hospital mortality, length of stay (LOS), hospital costs, and home discharge rates. Multivariable regression analyses adjusted for patient demographics, comorbidities, and hospital characteristics. Results: A total of 5,007 FG and 24,782 non-perineal NSTI patients were identified. Crude in-hospital mortality rates were 5.8% for FG and 5.4% for non-perineal NSTIs, with stable trends observed over five years. After adjustment, no significant difference in mortality was observed (adjusted odds ratio [aOR]: 1.04; 95% CI: 0.90–1.20). However, FG was associated with longer LOS (adjusted mean difference: 1.99 days; 95% CI: 1.53–2.46) and higher hospital costs ($37,809 higher; 95% CI: $29,540–$46,077). Home discharge rates were similar between groups (aOR: 0.97; 95% CI: 0.89–1.05). Discussion: Despite similar mortality rates, FG hospitalizations were associated with increased LOS and higher costs compared to non-perineal NSTIs. These findings may suggest potential nationwide disparities in FG care quality, particularly outside specialized referral centers. Further research is needed to understand if standardized care pathways tailored to FG may optimize management and reduce resource utilization.
- 30 May 2025
- Published by Wiley in Journal of the American Geriatrics Society
- Vol. 73 (7) , 2316-2318
Unveiling Hidden Abuse of Older Adults: A Case Study on the Critical Role of Clinical Vigilance and Skin Examinations
- 30 May 2025
- Published by Wiley in Journal of the American Geriatrics Society
- Vol. 73 (7) , 2316-2318
Unveiling Hidden Abuse of Older Adults: A Case Study on the Critical Role of Clinical Vigilance and Skin Examinations
- 27 December 2024
- Published by Springer Nature
- p. 61-79
Geriatric 5Ms in Patients with Kidney Disease
- 15 October 2024
- Published by SAGE Publications in Journal of Palliative Care
- Vol. 40 (2) , 120-128
Trends, Predictors, and Potential Disparities of Palliative Care Utilization Among Older Adults With Hip Fractures: A Retrospective Analysis of the National Inpatient Sample, 2016–2020
Objectives: To assess the trends in palliative care consultation utilization and identify associated factors among older adults hospitalized with hip fractures in the United States between 2016 and 2020. Methods: We conducted a retrospective cohort study using data from the National Inpatient Sample from 2016 to 2020. The study included patients aged 65 and older admitted to hospital with a primary diagnosis of hip fracture. We identified palliative care consultations using ICD-10 code Z51.5. Multivariate logistic regression analyses were performed to identify predictors of palliative care utilization, adjusting for demographics, clinical variables, and hospital characteristics. Results: A total of 293,749 admissions for hip fractures were identified, of which 9546 (3.2%) had palliative care consultations. A consistent upward trend was seen in the proportion of patients receiving palliative care consultations across all fracture types. Patients of color (Black: odds ratio [OR] = 0.73, 95% confidence interval [CI]: 0.65-0.83; Hispanic: OR = 0.67, 95% CI: 0.60-0.75 compared to White), those in lower-income quartiles (lowest: OR = 0.81, 95% CI: 0.76-0.87 compared to highest), smaller hospital size (OR = 0.84, 95% CI: 0.79-0.89 compared to large), and rural hospital (OR = 0.63, 95% Cl: 0.59-0.68, compared to urban teaching) were associated with fewer palliative care consultations. Conclusion: The number of palliative care consultations during hip fracture hospital admission was low, although the proportion increased over time. Significant disparities were observed and further research should explore barriers to palliative care access and develop strategies to enhance its delivery across diverse healthcare settings.
- 15 October 2024
- Published by SAGE Publications in Journal of Palliative Care
- Vol. 40 (2) , 120-128
Trends, Predictors, and Potential Disparities of Palliative Care Utilization Among Older Adults With Hip Fractures: A Retrospective Analysis of the National Inpatient Sample, 2016–2020
Objectives: To assess the trends in palliative care consultation utilization and identify associated factors among older adults hospitalized with hip fractures in the United States between 2016 and 2020. Methods: We conducted a retrospective cohort study using data from the National Inpatient Sample from 2016 to 2020. The study included patients aged 65 and older admitted to hospital with a primary diagnosis of hip fracture. We identified palliative care consultations using ICD-10 code Z51.5. Multivariate logistic regression analyses were performed to identify predictors of palliative care utilization, adjusting for demographics, clinical variables, and hospital characteristics. Results: A total of 293,749 admissions for hip fractures were identified, of which 9546 (3.2%) had palliative care consultations. A consistent upward trend was seen in the proportion of patients receiving palliative care consultations across all fracture types. Patients of color (Black: odds ratio [OR] = 0.73, 95% confidence interval [CI]: 0.65-0.83; Hispanic: OR = 0.67, 95% CI: 0.60-0.75 compared to White), those in lower-income quartiles (lowest: OR = 0.81, 95% CI: 0.76-0.87 compared to highest), smaller hospital size (OR = 0.84, 95% CI: 0.79-0.89 compared to large), and rural hospital (OR = 0.63, 95% Cl: 0.59-0.68, compared to urban teaching) were associated with fewer palliative care consultations. Conclusion: The number of palliative care consultations during hip fracture hospital admission was low, although the proportion increased over time. Significant disparities were observed and further research should explore barriers to palliative care access and develop strategies to enhance its delivery across diverse healthcare settings.
- Open Access
- 13 August 2024
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 10, e59133
Educational Utility of Clinical Vignettes Generated in Japanese by ChatGPT-4: Mixed Methods Study
Background: Evaluating the accuracy and educational utility of artificial intelligence–generated medical cases, especially those produced by large language models such as ChatGPT-4 (developed by OpenAI), is crucial yet underexplored. Objective: This study aimed to assess the educational utility of ChatGPT-4–generated clinical vignettes and their applicability in educational settings. Methods: Using a convergent mixed methods design, a web-based survey was conducted from January 8 to 28, 2024, to evaluate 18 medical cases generated by ChatGPT-4 in Japanese. In the survey, 6 main question items were used to evaluate the quality of the generated clinical vignettes and their educational utility, which are information quality, information accuracy, educational usefulness, clinical match, terminology accuracy (TA), and diagnosis difficulty. Feedback was solicited from physicians specializing in general internal medicine or general medicine and experienced in medical education. Chi-square and Mann-Whitney U tests were performed to identify differences among cases, and linear regression was used to examine trends associated with physicians’ experience. Thematic analysis of qualitative feedback was performed to identify areas for improvement and confirm the educational utility of the cases. Results: Of the 73 invited participants, 71 (97%) responded. The respondents, primarily male (64/71, 90%), spanned a broad range of practice years (from 1976 to 2017) and represented diverse hospital sizes throughout Japan. The majority deemed the information quality (mean 0.77, 95% CI 0.75-0.79) and information accuracy (mean 0.68, 95% CI 0.65-0.71) to be satisfactory, with these responses being based on binary data. The average scores assigned were 3.55 (95% CI 3.49-3.60) for educational usefulness, 3.70 (95% CI 3.65-3.75) for clinical match, 3.49 (95% CI 3.44-3.55) for TA, and 2.34 (95% CI 2.28-2.40) for diagnosis difficulty, based on a 5-point Likert scale. Statistical analysis showed significant variability in content quality and relevance across the cases (P<.001 after Bonferroni correction). Participants suggested improvements in generating physical findings, using natural language, and enhancing medical TA. The thematic analysis highlighted the need for clearer documentation, clinical information consistency, content relevance, and patient-centered case presentations. Conclusions: ChatGPT-4–generated medical cases written in Japanese possess considerable potential as resources in medical education, with recognized adequacy in quality and accuracy. Nevertheless, there is a notable need for enhancements in the precision and realism of case details. This study emphasizes ChatGPT-4’s value as an adjunctive educational tool in the medical field, requiring expert oversight for optimal application.
- 13 August 2024
- Published by JMIR Publications Inc. in JMIR Medical Education
- Vol. 10, e59133
Open AccessEducational Utility of Clinical Vignettes Generated in Japanese by ChatGPT-4: Mixed Methods Study
Background: Evaluating the accuracy and educational utility of artificial intelligence–generated medical cases, especially those produced by large language models such as ChatGPT-4 (developed by OpenAI), is crucial yet underexplored. Objective: This study aimed to assess the educational utility of ChatGPT-4–generated clinical vignettes and their applicability in educational settings. Methods: Using a convergent mixed methods design, a web-based survey was conducted from January 8 to 28, 2024, to evaluate 18 medical cases generated by ChatGPT-4 in Japanese. In the survey, 6 main question items were used to evaluate the quality of the generated clinical vignettes and their educational utility, which are information quality, information accuracy, educational usefulness, clinical match, terminology accuracy (TA), and diagnosis difficulty. Feedback was solicited from physicians specializing in general internal medicine or general medicine and experienced in medical education. Chi-square and Mann-Whitney U tests were performed to identify differences among cases, and linear regression was used to examine trends associated with physicians’ experience. Thematic analysis of qualitative feedback was performed to identify areas for improvement and confirm the educational utility of the cases. Results: Of the 73 invited participants, 71 (97%) responded. The respondents, primarily male (64/71, 90%), spanned a broad range of practice years (from 1976 to 2017) and represented diverse hospital sizes throughout Japan. The majority deemed the information quality (mean 0.77, 95% CI 0.75-0.79) and information accuracy (mean 0.68, 95% CI 0.65-0.71) to be satisfactory, with these responses being based on binary data. The average scores assigned were 3.55 (95% CI 3.49-3.60) for educational usefulness, 3.70 (95% CI 3.65-3.75) for clinical match, 3.49 (95% CI 3.44-3.55) for TA, and 2.34 (95% CI 2.28-2.40) for diagnosis difficulty, based on a 5-point Likert scale. Statistical analysis showed significant variability in content quality and relevance across the cases (P<.001 after Bonferroni correction). Participants suggested improvements in generating physical findings, using natural language, and enhancing medical TA. The thematic analysis highlighted the need for clearer documentation, clinical information consistency, content relevance, and patient-centered case presentations. Conclusions: ChatGPT-4–generated medical cases written in Japanese possess considerable potential as resources in medical education, with recognized adequacy in quality and accuracy. Nevertheless, there is a notable need for enhancements in the precision and realism of case details. This study emphasizes ChatGPT-4’s value as an adjunctive educational tool in the medical field, requiring expert oversight for optimal application.