Abstract
Purpose To propose a novel, standardized, and safety-centered tool for Large Language Models (LLMs) evaluation. Methods The Medical Evaluation of Large Language Model Answers Questionnaire (MELMA-Q) was developed as a 30-item clinician-rated instrument spanning seven domains: Medical Accuracy/Groundedness, Clinical Reasoning/Management, Safety/Ethics/Trustworthiness, Linguistic Quality/Semantic Fidelity, Understandability/Literacy Adaptation, Usefulness/Decision Support, and Performance/Answer Behavior. The MELMA Clinical Acceptability Framework (MELMA-CAF) is a two-tier system that incorporates a non-compensatory safety gate and weighted scoring. Five standardized otolaryngology scenarios were posed to three LLMs (ChatGPT 5.2, Gemini Flash 3, DeepSeek v3.2), generating 15 responses, which were independently scored by five blinded ENT specialists. A web-based implementation (MELMA-W) operationalized rubric-based scoring and was compared with clinician ratings. Results All responses passed Tier A safety screening. Mean total MELMA-Q scores ranged from 72.4 to 85.6 across models; inter-rater reliability was excellent (ICC 0.89; 95% CI 0.84–0.93). MELMA-W validation using paired model × domain observations showed systematically higher clinician scores (bias of 0.804 Likert points). Conclusions MELMA-Q and MELMA-W provide a structured, safety-centered pilot framework for evaluating LLM-generated medical responses in otolaryngology; however, broader validation across larger datasets, additional raters, and other clinical specialties remains required.

This publication has 35 references indexed in Scilit: