Evaluating the quality of medical content on YouTube using large language models
Authors:
Open Access
- 22 March 2025
- journal article
- research article
- Published by Springer Nature in Scientific Reports
- Vol. 15 (1) , 1-12
Abstract
YouTube has become a dominant source of medical information and health-related decision-making. Yet, many videos on this platform contain inaccurate or biased information. Although expert reviews could help mitigate this situation, the vast number of daily uploads makes this solution impractical. In this study, we explored the potential of Large Language Models (LLMs) to assess the quality of medical content on YouTube. We collected a set of videos previously evaluated by experts and prompted twenty models to rate their quality using the DISCERN instrument. We then analyzed the inter-rater agreement between the language models’ and experts’ ratings using Brennan–Prediger’s (BP) Kappa. We found that LLMs exhibited a wide range of inter-rater agreements with the experts (ranging from −1.10 to 0.82). All models tended to give higher scores than the human experts. The agreement on individual questions tended to be lower, with some questions showing significant disagreement between models and experts. Including scoring guidelines in the prompt has improved model performance. We conclude that some LLMs are capable of evaluating the quality of medical videos. If used as stand-alone expert systems or embedded into traditional recommender systems, these models can mitigate the quality issue of health-related online videos.Keywords
This publication has 42 references indexed in Scilit:
- Evaluating the Effectiveness of advanced large language models in medical Knowledge: A Comparative study using Japanese national medical examinationInternational Journal of Medical Informatics, 2024
- Diagnostic performances of GPT-4o, Claude 3 Opus, and Gemini 1.5 Pro in “Diagnosis Please” casesJapanese Journal of Radiology, 2024
- Large Language Models for Binary Health-Related Question Answering: A Zero- and Few-Shot Evaluation2024
- Large language models and automated essay scoring of English language learner writing: Insights into validity and reliabilityComputers and Education: Artificial Intelligence, 2024
- Automatic assessment of text-based responses in post-secondary education: A systematic reviewComputers and Education: Artificial Intelligence, 2024
- Using LLMs to bring evidence-based feedback into the classroom: AI-generated feedback increases secondary students’ text revision, motivation, and positive emotionsComputers and Education: Artificial Intelligence, 2023
- YouTube and pudendal neuralgia: Is it a good source of information for patients?Clinical Neurology and Neurosurgery, 2023
- Perceptual image quality assessment: a surveyScience China Information Sciences, 2020
- The effect of rater accountability on performance ratings: A meta-analytic reviewHuman Resource Management Review, 2017
- Assessing, Controlling, and Assuring the Quality of Medical Information on the InternetJAMA, 1997