Analyzing the Performance of ChatGPT About Osteoporosis
Author:
Open Access
- 25 September 2023
- journal article
- research article
- Published by Springer Nature in Cureus
- Vol. 15 (9) , e45890
Abstract
Introduction: This study evaluates the knowledge of ChatGPT about osteoporosis. Methods: Osteoporosis-related frequently asked questions (FAQs) created by examining the websites frequently visited by patients, the official websites of hospitals, and social media. Questions based on these scientific data have been prepared in accordance with National Osteoporosis Guideline Group guides. Rater scored all ChatGPT answers between 1 and 4 (1 stated that the information was completely correct, 2 stated that the information was correct but insufficient, 3 stated that although some of the information was correct, there was incorrect information in the answer, and 4 stated that the answer consisted of completely incorrect information). The reproducibility of ChatGPT responses on osteoporosis was assessed by asking each question twice. The repeatability of the ChatGPT answer was considered as getting the same score twice. Results: ChatGPT responded to 72 FAQs with an accuracy rate of 80.6%. The highest accuracy in ChatGPT's answers about osteoporosis was in the prevention category, 91.7%, and in the general knowledge category, 85.8%. Only 19 of the 31 (61.3%) questions prepared according to the National Osteoporosis Guideline Group guidelines were answered correctly by ChatGPT, and two answers (6.4%) were categorized as grade 4. The reproducibility rate of ChatGPT answers on 72 FAQs was 86.1% and the reproducibility rate of ChatGPT answers on National Osteoporosis Guideline Group guidelines was 83.9%. Conclusion: Present study outcomes for the first time showed that ChatGPT provided adequate answers to more than 80% of FAQs about osteoporosis. However, the accuracy of ChatGPT’s answers to inquiries based on National Osteoporosis Guideline Group guidelines was decreased to 61.3%.Keywords
This publication has 13 references indexed in Scilit:
- Evaluating the Performance of ChatGPT in OphthalmologyOphthalmology Science, 2023
- Evaluation of ChatGPT's Capabilities in Medical Report GenerationCureus, 2023
- Is ChatGPT an Evidence-based Doctor?European Urology, 2023
- How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge AssessmentJMIR Medical Education, 2023
- Global, regional prevalence, and risk factors of osteoporosis according to the World Health Organization diagnostic criteria: a systematic review and meta-analysisOsteoporosis International, 2022
- Global prevalence of osteoporosis among the world older adults: a comprehensive systematic review and meta-analysisJournal of Orthopaedic Surgery and Research, 2021
- Healthcare information on YouTube: Pregnancy and COVID‐19International Journal of Gynecology & Obstetrics, 2020
- Prevalence and contributing factors of osteoporosis in the elderly over 70 years old: an epidemiological study of several community health centers in ShanghaiAnnals of Palliative Medicine, 2020
- ‘Fake News’ in urology: evaluating the accuracy of articles shared on social media in genitourinary malignanciesBJU International, 2019
- Prevalence Of Osteoporosis In Women: Variation With Skeletal Site Of Measurement Of Bone Mineral DensityEndocrine Practice, 2000