Evaluation of large language model-based chatbots in hearing aid consultations: A comparative study of experts and non-experts
Open Access
- 1 February 2026
- journal article
- research article
- Published by SAGE Publications in Digital Health
- Vol. 12
Abstract
Objective: Hearing loss patients frequently seek reliable information and guidance regarding hearing aids. This study evaluated the clinical suitability and user preference of general-purpose large language models (LLMs) in hearing aid consultations to identify the current limitations of generic models and the necessity for domain-specific solutions. Methods: A dataset of 1,632 real-world inquiries was refined into 33 representative items across seven audiological categories through a blind review and unanimous consensus by a panel of experienced otolaryngologists and audiologists. Three LLMs (ChatGPT, HyperCLOVA X, and MaumAI) generated responses to these items. To establish a clinical reference standard, the author panel used a unanimous consensus protocol to select the single most accurate response from ten repetitions per model. The accuracy and preference of these responses were evaluated on a 3-point scale by 20 hearing healthcare experts and 33 non-experts (patients/guardians). Results: Both expert and non-expert groups showed a statistically significant preference for ChatGPT (LLM 1) regarding overall accuracy and preference. However, consistency within and between the two groups was notably low (Kendall’s W < 0.07), revealing a critical ‘perspective gap’. While experts prioritized clinical precision and technical specificity, non-experts favored responses that were practical, empathetic, and easy to understand. Conclusions: Current general-purpose LLMs struggle to simultaneously satisfy rigorous medical standards and user-centric readability, highlighting the limitations of a “one-size-fits-all” approach. To effectively bridge the health literacy gap, future research should focus on developing a customized Small Large Language Model (sLLM) trained on curated audiological data. Such a domain-specific model is essential to provide both medically reliable and patient-friendly digital hearing healthcare.Keywords
This publication has 30 references indexed in Scilit:
- Comparing ChatGPT and GPT-4 performance in USMLE soft skill assessmentsScientific Reports, 2023
- Developing chatbot simulations of patient interviews for history taking educationInternational Dental Journal, 2023
- Battle of the (Chat)Bots: Comparing Large Language Models to Practice Guidelines for Transfusion-Associated Graft-Versus-Host Disease PreventionTransfusion Medicine Reviews, 2023
- Will artificial intelligence chatbots replace clinical pharmacologists? An exploratory study in clinical practiceEuropean Journal of Clinical Pharmacology, 2023
- Large language models encode clinical knowledgeNature, 2023
- Medical visual question answering: A surveyArtificial Intelligence in Medicine, 2023
- Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge AssessmentJAMA Ophthalmology, 2023
- AI-Generated Medical Advice—GPT and BeyondJAMA, 2023
- Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for MedicineNew England Journal of Medicine, 2023
- Applied tele-audiology research in clinical practice during the past decade: a scoping reviewInternational Journal of Audiology, 2020