A Multiassessment and Multiprofessional Agents Approach for Medical Chatbot Risk Estimation: Development and Evaluation Study (Preprint)
Preprint
- 15 July 2025
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
BACKGROUND Assessing chatbot responses across 3 domains—medical, ethical, and legal—is essential to ensuring the safe use of artificial intelligence in health care. Although advancements in the use of large language models (LLMs) show significant improvements in evaluating question-answer datasets, such as multiple-choice medical exams, existing systems use general LLMs without incorporating specialized domain knowledge. They rely on standardized instructions without integrating real-world information, and ensemble methods such as majority voting fail to resolve disagreements among agents, resulting in misclassification and challenges in risk assessment. OBJECTIVE This study aims to design, develop, and evaluate a synergistic approach for assessing risks associated with chatbot responses using multiassessment (MA) and multiprofessional agents (MPAs). METHODS We designed and developed an approach consisting of MA and MPA, specifically initial assessment (MA1), which internalizes 3 roles and provides an initial risk estimation, and final assessment (MA3), which aims to reach a final consensus based on the previous assessments (MA1 and MA2), with each using 1 LLM. The verification assessment (MA2) incorporates an MPA or role-based LLM specialized agents for each risk domain (medical, ethical, and legal). We evaluated the proposed approach using the MedNLP-CHAT (Medical Natural Language Processing for AI Chat) corpus (N=226; 100 train, 126 test), covering baseline, enhanced prompt, embedding-based search, and retrieval-augmented generation (RAG). Primary metrics included macro F1-score and joint accuracy to evaluate system performance, along with CI and paired macro F1-score difference (Δ) as supporting metrics to assess the approach’s effectiveness. RESULTS The MA-MPA framework integrated with RAG achieved the highest average macro F1-score of 0.800 across risk domains and a joint accuracy of 76 (60.3%) correct predictions across all risk domains out of 126 question-answer pairs, with notable improvements over the best reported Eighteenth NII Testbeds and Community for Information Access Research Project (NTCIR-18) MedNLP-CHAT systems in the ethical (+0.252) and legal (+0.096) risk domains, while the medical domain showed a modest increase of +0.070. The MA approach contributed the largest gains, particularly from MA1 to MA2, with paired macro F1-score gains ranging from +0.176 to +0.214 across systems. The MPA approach performed better when integrated with MA and external knowledge, with paired bootstrap estimates showing a gain of +0.037 (95% CI 0.003-0.074) over baseline; however, joint accuracy gains were not evident (95% CI –2.9% to 7.7%), and gains relative to the enhanced prompt were small. Notably, MA alone achieved higher joint accuracy than RAG (62.7% vs 60.3%), indicating a metric-specific trade-off rather than consistent superiority across all metrics. CONCLUSIONS The MA-MPA approach shows potential for improving risk estimation in chatbot responses. The results suggest that the framework is particularly useful for enhancing balanced overall performance, especially when combined with external knowledge, although the medical risk domain remains challenging. Furthermore, more specialized LLMs may further improve contextually grounded risk estimation.Keywords
All Related Versions
- Published version: JMIR Medical Informatics, 14, e80416.
This publication has 12 references indexed in Scilit:
- Evaluating large language models and agents in healthcare: key challenges in clinical applicationsIntelligent Medicine, 2025
- Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross‐Sectional StudyArthritis & Rheumatology, 2024
- Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessmentsScientific Reports, 2023
- Performance of ChatGPT and GPT-4 on Neurosurgery Written Board ExaminationsNeurosurgery, 2023
- Large language models encode clinical knowledgeNature, 2023
- The emergent role of artificial intelligence, natural learning processing, and large language models in higher education and researchResearch in Social and Administrative Pharmacy, 2023
- Analysis of large-language model versus human performance for genetics questionsEuropean Journal of Human Genetics, 2023
- Assessing the Accuracy of Responses by the Language Model ChatGPT to Questions Regarding Bariatric SurgeryObesity Surgery, 2023
- Delivering Health Intelligence For Healthcare Services2019
- Health monitoring and tracking system for soldiers using Internet of Things(IoT)2017