AI-generated clinical summaries: errors and susceptibility to speech and speaker variability
Authors:
Open Access
- 24 April 2026
- journal article
- Published by BMJ in BMJ Health & Care Informatics
- Vol. 33 (1) , e101918
Abstract
Objectives: To evaluate whether variability in patients’ communication style (personality), international English-accents (human and synthetic) and speech impairments affects the accuracy of a Clinical AI Scribe (CAIS) and identify where performance degrades to inform pre-deployment validation and monitoring. Methods: We conducted simulated primary-care consultations using trained actors. For personality types, four scenarios were enacted, each with five patient-personality types. For accents, transcripts of consultations were used to generate combinations of seven accents across five scenarios. The CAIS produced summaries that were compared with transcripts, and errors classified as omissions, factual inaccuracies or hallucinations. For speech impairments, public recordings representing five profiles were transcribed and word-recognition accuracy calculated. Results: Personality types showed no statistically significant differences in errors (all p>0.05). Extraversion had the highest total errors (median 3.5). Across accents, comparisons were non-significant for both patient and doctor voices (patients: p=0.851; doctors: p=0.980). Omissions predominated, with low rates of hallucinations and factual inaccuracies. Omissions were slightly higher for Chinese-accented and Indian-accented doctors (both medians 3.0). Conversely, speech impairments differed: cleft palate and vowel disorders were near-perfect, whereas phonological impairment markedly reduced recognition (p<0.001). Discussion: Operationally, CAIS deployment should include clinician-in-the-loop verification, subgroup performance monitoring (accents, impairments) and predefined ‘switch-off’ criteria for severe phonological patterns. High-quality synthetic voices are a pragmatic proxy for accent testing when balanced corpora are unavailable. Conclusions: Under controlled conditions CAIS performance was broadly stable across communication styles and most accents, but vulnerable to specific speech characteristics, particularly phonological impairment, in this single-system simulation study.Keywords
Funding Information
- NHS England South West, via Bristol, North Somerset, and South Gloucestershire Integrated Care Board (11066090)
This publication has 24 references indexed in Scilit:
- Beyond human ears: navigating the uncharted risks of AI scribes in clinical practicenpj Digital Medicine, 2025
- An evaluation framework for ambient digital scribing tools in clinical applicationsnpj Digital Medicine, 2025
- A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisationnpj Digital Medicine, 2025
- Ambient Artificial Intelligence Scribes: Learnings after 1 Year and over 2.5 Million UsesNEJM Catalyst, 2025
- Artificial Intelligence Scribe and Large Language Model Technology in Healthcare Documentation: Advantages, Limitations, and RecommendationsPlastic and Reconstructive Surgery - Global Open, 2025
- Towards inclusive automatic speech recognitionComputer Speech & Language, 2023
- Medical Malpractice Trends: Errors in Automated Speech RecognitionJournal of Medical Systems, 2018
- Analysis of Errors in Dictated Clinical Documents Assisted by Speech Recognition Software and Professional TranscriptionistsJAMA Network Open, 2018
- The Oxford Handbook of the Five Factor Model2015
- Differences between acoustic characteristics of spontaneous and read speech and their effects on speech recognition performanceComputer Speech & Language, 2008