“MELMA” in otolaryngology: Medical evaluation of large language model answers. Clinician-rated scoring (MELMA-Q) and web-based auditing (MELMA-W) novel tools for AI assessment
Top Cited Papers
- 22 June 2026
- journal article
- other
- Published by Springer Nature in European Archives of Oto-Rhino-Laryngology
- p. 1-16
Abstract
Purpose To propose a novel, standardized, and safety-centered tool for Large Language Models (LLMs) evaluation. Methods The Medical Evaluation of Large Language Model Answers Questionnaire (MELMA-Q) was developed as a 30-item clinician-rated instrument spanning seven domains: Medical Accuracy/Groundedness, Clinical Reasoning/Management, Safety/Ethics/Trustworthiness, Linguistic Quality/Semantic Fidelity, Understandability/Literacy Adaptation, Usefulness/Decision Support, and Performance/Answer Behavior. The MELMA Clinical Acceptability Framework (MELMA-CAF) is a two-tier system that incorporates a non-compensatory safety gate and weighted scoring. Five standardized otolaryngology scenarios were posed to three LLMs (ChatGPT 5.2, Gemini Flash 3, DeepSeek v3.2), generating 15 responses, which were independently scored by five blinded ENT specialists. A web-based implementation (MELMA-W) operationalized rubric-based scoring and was compared with clinician ratings. Results All responses passed Tier A safety screening. Mean total MELMA-Q scores ranged from 72.4 to 85.6 across models; inter-rater reliability was excellent (ICC 0.89; 95% CI 0.84–0.93). MELMA-W validation using paired model × domain observations showed systematically higher clinician scores (bias of 0.804 Likert points). Conclusions MELMA-Q and MELMA-W provide a structured, safety-centered pilot framework for evaluating LLM-generated medical responses in otolaryngology; however, broader validation across larger datasets, additional raters, and other clinical specialties remains required.Keywords
This publication has 35 references indexed in Scilit:
- Testing and Evaluation of Health Care Applications of Large Language ModelsJAMA, 2025
- Medical large language models are vulnerable to data-poisoning attacksNature Medicine, 2025
- Why we need to be careful with LLMs in medicineFrontiers in Medicine, 2024
- Validation of the Quality Analysis of Medical Artificial Intelligence (QAMAI) tool: a new tool to assess the quality of health information provided by AI platformsEuropean Archives of Oto-Rhino-Laryngology, 2024
- Pilot Testing of a Tool to Standardize the Assessment of the Quality of Health Information Generated by Artificial Intelligence-Based ModelsCureus, 2023
- Prompt Engineering as an Important Emerging Skill for Medical Professionals: TutorialJournal of Medical Internet Research, 2023
- Validity and reliability of an instrument evaluating the performance of intelligent chatbot: the Artificial Intelligence Performance Instrument (AIPI)European Archives of Oto-Rhino-Laryngology, 2023
- Semantic coherence markers: The contribution of perplexity metricsArtificial Intelligence in Medicine, 2022
- Leniency Bias in Performance Ratings: The Big-Five CorrelatesFrontiers in Psychology, 2017
- STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENTThe Lancet, 1986