Performance of Large Language Models Under Input Variability in Health Care Applications: Dataset Development and Experimental Evaluation
Open Access
- 20 February 2026
- journal article
- Published by JMIR Publications Inc. in JMIR AI
- Vol. 5, e83640
Abstract
Background: Large language models (LLMs) are increasingly integrated into health care, where they contribute to patient care, administrative efficiency, and clinical decision-making. Despite their growing role, the ability of LLMs to handle imperfect inputs remains underexplored. These imperfections, which are common in clinical documentation and patient-generated data, may affect model reliability. Objective: This study investigates the impact of input perturbations on LLM performance across three dimensions: (1) overall effectiveness in different health-related applications, (2) comparative effects of different types and levels of perturbations, and (3) differential impact of perturbations on health-related terms versus non–health-related terms. Methods: We systematically evaluate 3 LLMs on 3 health-related tasks using a novel dataset containing 3 types of human-like variations (redaction, homophones, and typographical errors) at different perturbation levels. Results: Contrary to expectations, LLMs demonstrate notable robustness to common variations, and in more than half of the cases (151/270, 55.92%), the performance was stable or improved. In some cases (38/270, 14.07%), variations resulted in an increased performance, especially when dealing with lower perturbation levels. Redactions, often stemming from privacy concerns or cognitive lapses, are more detrimental than other variations. Conclusions: Our findings highlight the need for health care applications powered by LLMs to be designed with input variability in mind. Robustness to noisy or imperfect inputs is essential for maintaining reliability in real-world clinical settings, where data quality can vary widely. By identifying specific vulnerabilities and strengths, this study provides actionable insights for improving model resilience and guiding the development of safer, more effective artificial intelligence tools in health care. The accompanying dataset offers a valuable resource for further research into LLM performance under diverse conditions.Keywords
This publication has 27 references indexed in Scilit:
- A framework for human evaluation of large language models in healthcare derived from literature reviewnpj Digital Medicine, 2024
- Speech Sense Disambiguation: Tackling Homophone Ambiguity in End-to-End Speech Translation2024
- The Difficulties of Clinical NLP2023
- Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for MedicineNew England Journal of Medicine, 2023
- MLM-based typographical error correction of unstructured medical texts for named entity recognitionBMC Bioinformatics, 2022
- Explainable AI in Industry: Practical Challenges and Lessons Learned2020
- NICE, in Confidence: An Assessment of Redaction to Obscure Confidential Information in Single Technology Appraisals by the National Institute for Health and Care ExcellencePharmacoEconomics, 2019
- To Err is human? How typographical and orthographical errors affect perceptions of online reviewersComputers in Human Behavior, 2017
- Hurdles to Information Seeking: Spelling and Typographical Mistakes During Users' Online BehaviorJournal of the Association for Information Systems, 2006
- A Frequency-based Technique to Improve the Spelling Suggestion Rank in Medical QueriesJournal of the American Medical Informatics Association, 2004