Usefulness of Automatic Speech Recognition Assessment of Children With Speech Sound Disorders: Validation Study (Preprint)
Preprint
- 15 May 2024
- preprint
- Published by JMIR Publications Inc. in JMIR Preprints
Abstract
Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Speech sound disorders (SSDs) are common communication challenges in children, typically assessed by speech-language pathologists (SLPs) using standardized tools. However, traditional evaluation methods are time-intensive and prone to variability, raising concerns about reliability. Objective: This study aimed to compare the evaluation outcomes of SLPs and an automatic speech recognition (ASR) model using two standardized SSD assessments in South Korea, evaluating the ASR model’s performance. Methods: A fine-tuned wav2vec 2.0 XLS-R model, pretrained on 436,000 hours of adult voice data spanning 128 languages, was used. The model was further trained on 93.6 minutes of children’s voices with articulation errors to improve error detection. Participants included children referred to the Department of Rehabilitation Medicine at a general hospital in Incheon, South Korea, from August 19, 2022, to June 14, 2023. Two standardized assessments—the Assessment of Phonology and Articulation for Children (APAC) and the Urimal Test of Articulation and Phonology (U-TAP)—were used, with ASR transcriptions compared to SLP transcriptions. Results: This study included 30 children aged 3-7 years who were suspected of having SSDs. The phoneme error rates for the APAC and U-TAP were 8.42% (457/5430) and 8.91% (402/4514), respectively, indicating discrepancies between the ASR model and SLP transcriptions across all phonemes. Consonant error rates were 10.58% (327/3090) and 11.86% (331/2790) for the APAC and U-TAP, respectively. On average, there were 2.60 (SD 1.54) and 3.07 (SD 1.39) discrepancies per child for correctly produced phonemes, and 7.87 (SD 3.66) and 7.57 (SD 4.85) discrepancies per child for incorrectly produced phonemes, based on the APAC and U-TAP, respectively. The correlation between SLPs and the ASR model in terms of the percentage of consonants correct was excellent, with an intraclass correlation coefficient of 0.984 (95% CI 0.953-0.994) and 0.978 (95% CI 0.941-0.990) for the APAC and UTAP, respectively. The z scores between SLPs and ASR showed more pronounced differences with the APAC than the U-TAP, with 8 individuals showing discrepancies in the APAC compared to 2 in the U-TAP. Conclusions: The results demonstrate the potential of the ASR model in assessing children with SSDs. However, its performance varied based on phoneme or word characteristics, highlighting areas for refinement. Future research should include more diverse speech samples, clinical settings, and speech data to strengthen the model’s refinement and ensure broader clinical applicability.Keywords
All Related Versions
- Published version: Journal of Medical Internet Research, 27, e60520.
This publication has 19 references indexed in Scilit:
- Speech-Language Pathologists' Ratings of Speech Accuracy in Children With Speech Sound DisordersAmerican Journal of Speech-Language Pathology, 2022
- Automatic speech recognition: A primer for speech-language pathology researchersInternational Journal of Speech-Language Pathology, 2018
- Automated speech analysis tools for children’s speech production: A systematic literature reviewInternational Journal of Speech-Language Pathology, 2018
- Automatic analysis of pronunciations for children with speech sound disordersComputer Speech & Language, 2017
- Recognition test on highly newly robust Malay corpus based on statistical analysis for Malay articulation disorder2014
- How should children with speech sound disorders be classified? A review and critical evaluation of current classification systemsInternational Journal of Language & Communication Disorders, 2012
- Contribution of consonant versus vowel information to sentence intelligibility for young normal-hearing and elderly hearing-impaired listenersThe Journal of the Acoustical Society of America, 2007
- Speech-Language Pathologists' Assessment Practices for Children With Suspected Speech Sound Disorders: Results of a National SurveyAmerican Journal of Speech-Language Pathology, 2007
- Learning and teaching phonetic transcription for clinical purposesClinical Linguistics & Phonetics, 2002
- An Analysis of Perceptual Confusions Among Some English ConsonantsThe Journal of the Acoustical Society of America, 1955