Evaluation of large language models as a diagnostic tool for medical learners and clinicians using advanced prompting techniques
Open Access
- 1 August 2025
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 20 (8) , e0325803
Abstract
Background: Large language models (LLMs) have demonstrated capabilities in natural language processing and critical reasoning. Studies investigating their potential use as healthcare diagnostic tools have largely relied on proprietary models like ChatGPT and have not explored the application of advanced prompt engineering techniques. This study aims to evaluate the diagnostic accuracy of three open-source LLMs and the role of prompt engineering using clinical scenarios. Methods: We analyzed the performance of three open-source LLMs—llama-3.1-70b-versatile, llama-3.1-8b-instant, and mixtral-8x7b-32768—using advanced prompt engineering when answering Medscape Clinical Challenge questions. Responses were recorded and evaluated for correctness, accuracy, precision, specificity, and sensitivity. A sensitivity analysis was conducted presenting the three LLMs with basic prompting challenge questions and excluding cases with visual assets. Results were compared with previously published performance data on GPT-3.5. Results: Llama-3.1-70b-versatile, llama-3.1-8b-instant, and mixtral-8x7b-32768 achieved correct responses in 79%, 65%, and 62% of cases, respectively, outperforming GPT-3.5 (74%). Diagnostic accuracy, precision, sensitivity, and specificity responses all outperformed those previously reported for GPT-3.5. Results generated using advanced prompting strategies were superior to those based on basic prompting. Sensitivity analysis revealed similar trends when cases with visual assets were excluded. Discussion: Using advanced prompting techniques, LLMs can generate clinically accurate responses. The study highlights the limitations of proprietary models like ChatGPT, particularly in terms of accessibility and reproducibility due to version deprecation. Future research should employ prompt engineering techniques and prioritize the use of open-source models to ensure research replicability.Keywords
This publication has 22 references indexed in Scilit:
- Brève : Un générique pour la SEP dès 13 ansRevue Francophone des Laboratoires, 2024
- Evaluation of Rhinoplasty Information from ChatGPT, Gemini, and Claude for Readability and AccuracyAesthetic Plastic Surgery, 2024
- Urology consultants versus large language models: Potentials and hazards for medical advice in urologyBJUI Compass, 2024
- Harnessing artificial intelligence in bariatric surgery: comparative analysis of ChatGPT-4, Bing, and Bard in generating clinician-level bariatric surgery recommendationsSurgery for Obesity and Related Diseases, 2024
- Prompt engineering in consistency and reliability with the evidence-based guideline for LLMsnpj Digital Medicine, 2024
- ChatGPT in healthcare: A taxonomy and systematic reviewComputer Methods and Programs in Biomedicine, 2024
- Large language models in health care: Development, applications, and challengesHealth Care Science, 2023
- Large language models encode clinical knowledgeNature, 2023
- Big Data and Machine Learning in Health CareJAMA, 2018
- Natural language processing and its future in medicineAcademic Medicine, 1999