Diagnosis in Bytes: Comparing the Diagnostic Accuracy of Google and ChatGPT 3.5 as Diagnostic Support Tools
Preprint
Abstract
Objective Adopting digital technologies as diagnostic support tools in medicine is unquestionable. However, the accuracy in suggesting diagnoses remains controversial and underexplored. We aimed to evaluate and compare the diagnostic accuracy of two primary and accessible internet search tools: Google and ChatGPT 3.5. Method We used 60 clinical cases related to urological pathologies to evaluate both platforms. These cases were divided into two groups: one with common conditions (constructed from the most frequent symptoms, following EAU and UpToDate guidelines) and another with rare disorders - based on case reports published between 2022 and 2023 in Urology Case Reports. Each case was inputted into Google Search and ChatGPT 3.5, and the results were categorized as "correct diagnosis," "likely differential diagnosis," or "incorrect diagnosis." A team of researchers evaluated the responses blindly and randomly. Results In typical cases, Google achieved 53.3% accuracy, offering a likely differential diagnosis in 23.3% and errors in the rest. ChatGPT 3.5 exhibited superior performance, with 86.6% accuracy, and suggested a reasonable differential diagnosis in 13.3%, without mistakes. In rare cases, Google did not provide correct diagnoses but offered a likely differential diagnosis in 20%. ChatGPT 3.5 achieved 16.6% accuracy, with 50% differential diagnoses. Conclusion ChatGPT 3.5 demonstrated higher diagnostic accuracy than Google in both contexts. The platform showed acceptable accuracy in common cases; however, limitations in rare cases remained evident.Keywords
This publication has 20 references indexed in Scilit:
- Quality of information and appropriateness of ChatGPT outputs for urology patientsProstate Cancer and Prostatic Diseases, 2023
- Defining the Role of Large Language Models in Urologic Care and ResearchEuropean Urology Oncology, 2023
- ChatGPT and most frequent urological diseases: analysing the quality of information and potential risks for patientsWorld Journal of Urology, 2023
- Evaluating the performance of ChatGPT in answering questions related to pediatric urologyJournal of Pediatric Urology, 2023
- Caution! AI Bot Has Entered the Patient Chat: ChatGPT Has Limitations in Providing Accurate Urologic Healthcare AdviceUrology, 2023
- Artificial Intelligence and Machine Learning in Clinical Medicine, 2023New England Journal of Medicine, 2023
- Chatbots: History, technology, and applicationsMachine Learning with Applications, 2020
- Conversational agents in healthcare: a systematic reviewJournal of the American Medical Informatics Association, 2018
- Do PubMed and Google searches help medical students and young doctors reach the correct diagnosis? A pilot studyEuropean Journal of Internal Medicine, 2009
- Interpretation of Symptoms with a Data-Processing MachineArchives of internal medicine (1960), 1959