Evaluation of LLMs accuracy and consistency in the registered dietitian exam through prompt engineering and knowledge retrieval
Authors:
Top Cited Papers
Open Access
- 9 January 2025
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 15 (1) , 1-15
Abstract
Large language models (LLMs) are fundamentally transforming human-facing applications in the health and well-being domains: boosting patient engagement, accelerating clinical decision-making, and facilitating medical education. Although state-of-the-art LLMs have shown superior performance in several conversational applications, evaluations within nutrition and diet applications are still insufficient. In this paper, we propose to employ the Registered Dietitian (RD) exam to conduct a standard and comprehensive evaluation of state-of-the-art LLMs, GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro, assessing both accuracy and consistency in nutrition queries. Our evaluation includes 1050 RD exam questions encompassing several nutrition topics and proficiency levels. In addition, for the first time, we examine the impact of Zero-Shot (ZS), Chain of Thought (CoT), Chain of Thought with Self Consistency (CoT-SC), and Retrieval Augmented Prompting (RAP) on both accuracy and consistency of the responses. Our findings revealed that while these LLMs obtained acceptable overall performance, their results varied considerably with different prompts and question domains. GPT-4o with CoT-SC prompting outperformed the other approaches, whereas Gemini 1.5 Pro with ZS recorded the highest consistency. For GPT-4o and Claude 3.5, CoT improved the accuracy, and CoT-SC improved both accuracy and consistency. RAP was particularly effective for GPT-4o to answer Expert level questions. Consequently, choosing the appropriate LLM and prompting technique, tailored to the proficiency level and specific domain, can mitigate errors and potential risks in diet and nutrition chatbots.Keywords
All Related Versions
This publication has 45 references indexed in Scilit:
- The Impact of Multimodal Large Language Models on Health Care’s FutureJournal of Medical Internet Research, 2023
- The future landscape of large language models in medicineCommunications Medicine, 2023
- AI-Powered Renal Diet Support: Performance of ChatGPT, Bard AI, and Bing ChatClinics and Practice, 2023
- Accuracy, Reliability, and Comprehensibility of ChatGPT-Generated Medical Responses for Patients With Nonalcoholic Fatty Liver DiseaseClinical Gastroenterology and Hepatology, 2023
- Evaluating large language models on a highly-specialized topic, radiation oncology physicsFrontiers in Oncology, 2023
- Large language models in medicineNature Medicine, 2023
- Large language models encode clinical knowledgeNature, 2023
- The credibility of dietary advice formulated by ChatGPT: Robo-diets for people with food allergiesNutrition, 2023
- Evaluating the Performance of ChatGPT in OphthalmologyOphthalmology Science, 2023
- Computing Inter-Rater Reliability for Observational Data: An Overview and TutorialThe Quantitative Methods for Psychology, 2012