Clinical Evaluation of ChatGPT-5.3 Responses to Patient-Oriented Questions on Scoliosis: A Multidimensional Expert Analysis
Author:
Open Access
- 3 June 2026
- journal article
- research article
- Published by MDPI AG in Healthcare
- Vol. 14 (11) , 1563
Abstract
Background: The integration of large language models (LLMs) into healthcare has rapidly expanded, particularly in the domain of patient education. However, concerns remain regarding the accuracy, adequacy, and clinical safety of AI-generated medical information. This study aimed to systematically evaluate expert-perceived quality, content appropriateness, and potential clinical risk of responses generated by a GPT-5.3–based large language model to patient-oriented questions on scoliosis. Methods: A set of patient-oriented questions was developed based on common informational needs. Responses generated by the AI model were evaluated by a panel of experts using a multidimensional assessment framework, including general appropriateness, scientific accuracy, adequacy, and clarity. Content validity was assessed using the Content Validity Ratio (CVR) and Content Validity Index (CVI). In addition, clinical risk levels were categorized, and common issues in inappropriate responses were analyzed. CVR and CVI were used to quantify expert agreement regarding perceived content appropriateness, rather than to establish definitive factual correctness or guideline concordance. Results: All responses exceeded the predefined acceptable CVR threshold, and overall CVI values were high, indicating a high level of expert agreement regarding perceived response appropriateness. Clarity received the highest scores across all dimensions, whereas adequacy and scientific accuracy were relatively lower. Most responses were classified as harmless or low risk, and no high-risk responses were identified. The most frequently reported issue in inappropriate responses was insufficient information. Conclusions: Large language models may provide understandable and generally acceptable responses in a controlled expert-evaluation setting. However, these findings reflect expert-perceived appropriateness rather than definitive clinical validity or guideline concordance. AI-generated responses should therefore be used only as supportive educational tools under clinical oversight.Keywords
This publication has 24 references indexed in Scilit:
- Let’s chat (GPT) about adolescent idiopathic scoliosis: accuracy and reliability of chat responses to frequently asked questionsBMC Musculoskeletal Disorders, 2025
- The role of ChatGPT in improving orthopedic Patient education in low-resource settings across various orthopedic specialtiesThe Journal of Precision Medicine: Health and Disease, 2025
- Evaluating ChatGPT-4.0’s accuracy and potential in idiopathic scoliosis conservative treatment: a preliminary study on clarity, validity, and expert perceptionsEuropean Spine Journal, 2025
- The utility of ChatGPT in musculoskeletal imaging-related patient educationClinical Imaging, 2025
- Assessing the Accuracy and Reliability of AI-Generated Medical Responses: An Evaluation of the Chat-GPT ModelResearch Square, 2023
- Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language modelsPLOS Digital Health, 2023
- Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AIBMJ, 2022
- High-performance medicine: the convergence of human and artificial intelligenceNature Medicine, 2019
- A guide to deep learning in healthcareNature Medicine, 2019
- A QUANTITATIVE APPROACH TO CONTENT VALIDITY1Personnel Psychology, 1975