Accuracy, satisfaction, and impact of custom GPT in acquiring clinical knowledge: Potential for AI-assisted medical education
- 2 February 2025
- journal article
- Published by Taylor & Francis in Medical Teacher
- Vol. 47 (9) , 1502-1508
Abstract
Recent advancements in artificial intelligence (AI) have enabled the customization of large language models to address specific domains such as medical education. This study investigates the practical performance of a custom GPT model in enhancing clinical knowledge acquisition for medical students and physicians. A custom GPT was developed by incorporating the latest readily available teaching resources. Its accuracy in providing clinical knowledge was evaluated using a set of clinical questions, and responses were compared against established medical guidelines. Satisfaction was assessed through surveys involving medical students and physicians at different stages and from various types of hospitals. The impact of the custom GPT was further evaluated by comparing its role in facilitating clinical knowledge acquisition with traditional learning methods. The custom GPT demonstrated higher accuracy (83.6%) compared to general AI models (65.5%, 69.1%) and was comparable to a professionally developed AI (Glass Health, 83.6%). Residents reported the highest satisfaction compared to clerks and physicians, citing improved learning independence, motivation, and confidence (p < 0.05). Physicians, especially those from teaching hospitals, showed greater eagerness to develop a custom GPT compared to clerks and residents (p < 0.05). The impact analysis revealed that residents using the custom GPT achieved better test scores compared to those using traditional resources (p < 0.05), though fewer perfect scores were obtained. The custom GPT demonstrates significant promise as an innovative tool for advancing medical education, particularly for residents. Its capability to deliver accurate, tailored information complements traditional teaching methods, aiding educators in promoting personalized and consistent training. However, it is essential for both learners and educators to remain critical in evaluating AI-generated information. With continued development and thoughtful integration, AI tools like custom GPTs have the potential to significantly enhance the quality and accessibility of medical education.[Box: see text].Keywords
Funding Information
- Central South University
- National Natural Science Foundation of China (82090024)
- National Natural Science Foundation of China (82173877)
- Key Research and Development Program of Hunan Province (2021SK2015)
- Outstanding Youth Foundation of the Natural Science Foundation of Hunan Province (2022JJ10100)
- Central South University (2022jy136)
This publication has 9 references indexed in Scilit:
- Case-based MCQ generator: A custom ChatGPT based on published prompts in the literature for automatic item generationMedical Teacher, 2024
- Twelve tips on creating and using custom GPTs to enhance health professions educationMedical Teacher, 2024
- Evaluating performance of custom GPT in anesthesia practiceJournal of Clinical Anesthesia, 2023
- Performance of ChatGPT on Nephrology Test QuestionsClinical Journal of the American Society of Nephrology, 2023
- Assessing the Accuracy of an Online Chat-Based Artificial Intelligence Model in Providing Recommendations on Hypertension Management in Accordance With the 2017 American College of Cardiology/American Heart Association and 2018 European Society of Cardiology/European Society of Hypertension GuidelinesHypertension, 2023
- Assessing the Accuracy of ChatGPT on Core Questions in Glomerular DiseaseKidney International Reports, 2023
- Appropriateness of Cardiovascular Disease Prevention Recommendations Obtained From a Popular Online Chat-Based Artificial Intelligence ModelJAMA, 2023
- 2. Classification and Diagnosis of Diabetes:Standards of Care in Diabetes—2023Diabetes Care, 2022
- KDIGO 2022 Clinical Practice Guideline for Diabetes Management in Chronic Kidney DiseaseKidney International, 2022