Ambient AI Documentation in Mixed-Language Encounters: A Heuristic Evaluation of Reenacted Mandarin–English and Spanish–English Clinical Conversations
Authors:
Preprint
Abstract
Ambient AI documentation systems rely on automatic speech recognition to transcribe patient–provider conversations before generating clinical notes. However, little evidence exists on how these systems perform in mixed-language clinical encounters. We conducted a mixed-methods heuristic evaluation of an ambient AI documentation tool using 24 reenacted primary care conversations, including 12 Mandarin–English conversations developed from real-world encounter excerpts and 12 Spanish–English adapted counterparts. Quantitative analyses measured mixed error rate (MER) and code-switching. Overall MER was low, with a median of 4% and less variation in Spanish–English conversations, and 9% in Mandarin–English conversations, but with outliers reaching 67%. The system generally detected language switches reliably, although deletions occurred frequently in Mandarin–English transcripts at switch points. Qualitative analysis revealed transcription errors related to phonetic similarity, automatic translation, clinical terminology recognition, and language-specific challenges. These findings highlight considerations for improving ambient AI tools to support multilingual providers in delivering care for linguistically diverse populations.Keywords
This publication has 23 references indexed in Scilit:
- Evaluating ambient artificial intelligence documentation: effects on work efficiency, documentation burden, and patient-centered careJournal of the American Medical Informatics Association, 2025
- Ambient Documentation Technology in Clinician Experience of Documentation Burden and BurnoutJAMA Network Open, 2025
- Physician Perspectives on Ambient AI ScribesJAMA Network Open, 2025
- Ambient artificial intelligence scribes: utilization and impact on documentation timeJournal of the American Medical Informatics Association, 2024
- Ambient artificial intelligence scribes: physician burnout and perspectives on usability and documentation burdenJournal of the American Medical Informatics Association, 2024
- Challenges of Automatic Speech Recognition for medical interviews - research for Polish languageProcedia Computer Science, 2023
- Challenges of developing a digital scribe to reduce clinical documentation burdennpj Digital Medicine, 2019
- A Systematic Review of the Impact of Patient–Physician Non-English Language Concordance on Quality of Care and OutcomesJournal of General Internal Medicine, 2019
- The impact of limited English proficiency and physician language concordance on reports of clinical interactions among patients with diabetes: The DISTANCE studyPatient Education and Counseling, 2010
- Identification of Limited English Proficient Patients in Clinical CareJournal of General Internal Medicine, 2008