Reference decisions enhance LLM performance, amplified by source disclosure
- 1 April 2025
- journal article
- Published by SAGE Publications in Digital Health
- Vol. 11
Abstract
Objective: The rapid integration of large language models (LLMs) has propelled advancements in automated dialog technologies, improving the public's access to healthcare services. Drawing inspiration from the collaborative decision-making practices of medical professionals in complex cases, we investigated whether LLMs could enhance their diagnostic accuracy through interaction. Methods: An experimental study was conducted in China (September–December 2024) to investigate the impact of LLM-generated reference decisions and source disclosure on LLMs’ diagnostic performance. We used a Chinese clinical diagnostic task in a controlled comparative design, where three Chinese LLMs interpreted symptoms and conditions based on patient queries. LLMs’ outcomes were evaluated through accuracy and weighted F1 score metrics, with statistical analysis to determine significance. Results: Analysis of variance on LLMs’ diagnostic accuracy scores demonstrated that incorporating LLM-generated decisions as a reference significantly improved diagnostic outcomes, with source disclosure amplifying this improvement. Conclusion: Our findings underscore the potential of LLM collaboration in healthcare, offering strategies to refine response generation and decision-making across various applications.Keywords
Funding Information
- Research Grants Council of the Hong Kong Special Administrative Region, China (CityU 11507323)
This publication has 37 references indexed in Scilit:
- Diagnostic performances of GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro in “Diagnosis Please” casesJapanese Journal of Radiology, 2024
- ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reportsEuropean Radiology, 2023
- Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4Rheumatology International, 2023
- The Use of ChatGPT to Assist in Diagnosing Glaucoma Based on Clinical Case ReportsOphthalmology and Therapy, 2023
- Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care QuestionsJAMA Network Open, 2023
- Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric SurgeryObesity Surgery, 2023
- Aesthetic Surgery Advice and Counseling from Artificial Intelligence: A Rhinoplasty Consultation with ChatGPTAesthetic Plastic Surgery, 2023
- Living up to the chatbot hype: The influence of anthropomorphic design cues and communicative agency framing on conversational agent and company perceptionsComputers in Human Behavior, 2018
- Clustering ensemble methodInternational Journal of Machine Learning and Cybernetics, 2018
- Ensemble Methods in Machine Learning2000