The Impact of Acoustic and Informational Noise on AI-Generated Clinical Summaries
Preprint
Abstract
Objectives: To investigate the effect of the introduction of environmental noise, microphone identity & position, and informational noise on the accuracy of a commercial Clinical AI Scribe (CAIS). Design: Consultations on five medical conditions (memory loss, diarrhoea, headaches, skin rash, or prostate symptoms) were recorded. A ground-truth audio file was produced for a noise free consultation. This in turn produced a ground-truth clinical summary via the AI scribe, typically containing 30 facts. Different types of ‘noise’ at different levels were then investigated by (i) increasing the distance of the consultation from the microphone, and changing microphone type (ii) introducing background noise: heavy rain, baby crying, construction noise and a toddler at a range of levels between -10 to +5dB relative to the consultation audio and (iii) introducing informational noise: the medical consultations were modified to include additional conversations between the clinician and the patient to introduce irrelevant discussions (medical and non-medical, shorter and longer). The resulting 160 summaries were then compared with the corresponding ground-truth summaries and the errors introduced were classified and enumerated. Setting: Simulated primary care setting. Participants: The roles of the doctor and patient were played by actors, rotating between roles. Main outcome measures: Outcomes measured were the quantification of the degradation of accuracy of the clinical summary, i.e. omissions, hallucinations and inclusions, relative to the ground-truth summary, following the introduction of noise of all types at different levels. Results: Error rates and absolute error counts increased for all noise levels and noise types; the principal error being omissions. At 4.5 m from the consultation, for all microphones tested, all facts were omitted. At up to 2 m all microphones, except the laptop, only introduced a small number (<5) of omissions. For background noise, the type of noise was found to correlate with omissions – toddler and heavy rain were particularly deleterious, 7 and 15 omissions at 0 and +5dB respectively, i.e. when the noise loudness was comparable to or greater than that of the consultation. The CAIS was remarkably apt at rejecting additional informational noise. This was true for the addition of either short or long, medical or non-medical, information. Conclusions: The CAIS was very good at capturing a doctor-patient consultation and producing accurate clinical summaries, however error rate (especially omissions) increased notably when acoustic noise was introduced. The system was better able to handle informational noise. Summary Box: What is already known on this topic: The use of AI is rapidly increasing, with many commercial products available, both specific to clinical use and not. Commercial Clinical AI Scribes (generating clinical summaries from consultation audio) are being used in healthcare settings, sometimes with little guidance or knowledge of employers. These tools can operate well in perfect conditions, but there is limited data on nosier environments. What this study adds: The commercial Clinical AI Scribe only introduced a small number of errors when introduced to informational noise, however large numbers of errors (especially omissions) were caused by acoustic noise (dependant on the particular sound type and volume), especially with low-quality poorly-located microphones. Users of Clinical AI Scribes must be particularly vigilant in checking the summary when the environment contains certain deleterious background noises, even at lower volumes, to ensure errors are not introduced to the patient records.Keywords
All Related Versions
- Published version: BMJ Digital Health & AI, 1 (1), e000057.
This publication has 28 references indexed in Scilit:
- Large language models can support generation of standardized discharge summaries – A retrospective study utilizing ChatGPT-4 and electronic health recordsInternational Journal of Medical Informatics, 2024
- Use of an ambient artificial intelligence tool to improve quality of clinical documentationFuture Healthcare Journal, 2024
- Role of Artificial Intelligence in Medical Image Analysis: A Review of Current Trends and Future DirectionsJournal of Medical and Biological Engineering, 2024
- Adapted large language models can outperform medical experts in clinical text summarizationNature Medicine, 2024
- Aci-bench: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note GenerationScientific Data, 2023
- AI in health and medicineNature Medicine, 2022
- The effects of masking sound and signal-to-noise ratio on work performance in Chinese open-plan officesApplied Acoustics, 2021
- Challenges of developing a digital scribe to reduce clinical documentation burdennpj Digital Medicine, 2019
- Artificial intelligence in healthcareNature Biomedical Engineering, 2018
- How physician electronic health record screen sharing affects patient and doctor non-verbal communication in primary carePatient Education and Counseling, 2015