Cross-model disagreement as a reference-free signal for prioritizing human review in medical speech transcription
Open Access
- 1 July 2026
- journal article
- Published by Frontiers Media SA in Frontiers in Artificial Intelligence
- Vol. 9, 1829902
Abstract
Introduction: Ambient AI scribes generate transcripts at scale, but routine quality assurance is constrained by the absence of human-verified reference transcripts in most deployment settings. We evaluated whether disagreement among heterogeneous automatic speech recognition (ASR) systems can serve as an informative signal for localizing transcription uncertainty, using a public English-language medical-speech corpus rather than clinical encounter recordings. Methods: Eight commercial and open-source ASR systems were applied to 50 medical-education audio clips (8 h 14 min). Multi-model outputs were aligned, and a leave-one-out consensus procedure was used to score per-model agreement while reducing circularity. Results: Disagreement across models was sparse and localized: 72.1% of positions showed strong agreement (7-8 systems concordant), whereas only 2.5% were high-risk positions with minimal agreement (0-3 systems). Low-agreement regions were systematically enriched for meaning-bearing lexical differences, defined as lexical mismatches after excluding punctuation, contraction, numeric, and filler variation. A single-annotator human-corrected (HC) validation layer showed that transcription errors increased monotonically with decreasing agreement. At an illustrative post hoc threshold, flagging positions where six or fewer systems agreed selected 28.6% of tokens while recovering 93.7% of single-annotator HC-verified errors on this proxy corpus. Discussion: These findings suggest that cross-model disagreement may help focus human review on a small number of likely error-prone transcript regions. However, agreement among all systems does not guarantee correctness, because shared errors may remain undetected by this approach. Validation on real clinical encounter data is required before operational deployment.Keywords
This publication has 17 references indexed in Scilit:
- Policy brief: ambient AI scribes and the coding arms racenpj Digital Medicine, 2025
- Ambient AI Scribes in Clinical Practice: A Randomized TrialNEJM AI, 2025
- A Pragmatic Randomized Controlled Trial of Ambient Artificial Intelligence to Improve Health Practitioner Well-BeingNEJM AI, 2025
- Decoding disparities: evaluating automatic speech recognition system performance in transcribing Black and White patient verbal communication with nurses in home healthcareJAMIA Open, 2024
- Ambient Artificial Intelligence Scribes to Alleviate the Burden of Clinical DocumentationNEJM Catalyst, 2024
- A dataset of simulated patient-physician medical interviews with a focus on respiratory casesScientific Data, 2022
- Racial disparities in automated speech recognitionProceedings of the National Academy of Sciences of the United States of America, 2020
- Efficiency and safety of speech recognition for documentation in the electronic health recordJournal of the American Medical Informatics Association, 2017
- A post-processing system to yield reduced word error rates: Recognizer Output Voting Error Reduction (ROVER)2002
- Finding consensus in speech recognition: word error minimization and other applications of confusion networksComputer Speech & Language, 2000