Tae-Jin Yoon
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Tae-Jin Yoon has a total of 20 co-authors. Here are the top co-authors by number of shared publications:
- Seunghee Ha
Division of Speech Pathology and Audiology, Research Institute of Audiology and Speech Pathology, Hallym University, Chuncheon, Korea
7 shared publications - Jungmin So
Department of Computer Science and Engineering, Sogang University, Seoul, Korea
3 shared publications - Woojae Han
Division of Speech Pathology and Audiology, Research Institute of Audiology and Speech Pathology, Hallym University, Chuncheon, Korea
3 shared publications - Sangmin Park
Laboratory of Hearing and Technology, College of Natural Sciences, Hallym University, Chuncheon, Korea
2 shared publications - Sunmi Ma
Laboratory of Hearing and Technology, College of Natural Sciences, Hallym University, Chuncheon, Korea
2 shared publications - Yoonjung Kang
Department of Linguistics, University of Toronto 1 , Toronto, Ontario
2 shared publications
Latest Publications
- Open Access
- 31 October 2026
- Published by Korean Academy of Audiology in Audiology and Speech Research
- Vol. 22 (4) , 207-223
A Literature Review on Artificial Intelligence-Based Speech Intelligibility Assessment in Motor Speech Disorders
This scoping review aimed to examine the methodological characteristics and performance of artificial intelligence (AI)-based speech intelligibility assessment in individuals with acquired motor speech disorders. It also evaluated the clinical applicability of AI-based assessment by reviewing the relationship between AI-based assessments and human perceptual ratings. Four international databases (PubMed, IEEE Xplore, Web of Science, and ScienceDirect) and three Korean databases (KISS, DBPIA, and RISS) were searched up to March 2026. Fifteen studies were selected after applying predefined exclusion criteria. The selected studies were classified and synthesized descriptively without risk-of-bias assessment. 1) Reading tasks were the most frequently used speech task across the 15 included studies, regardless of the data collection method. 2) A wide range of automatic speech recognition systems (e.g., Google Cloud ASR API; Google LLC, Mountain View, CA, USA) and machine learning and deep learning models (e.g., bidirectional long-short term memory, convolutional neural networks, support vector machines, and linear regression) were used, with no single approach predominating. 3) Many studies reported strong correlations or high levels of agreement between AI-based assessments and human perceptual ratings, whereas some reported weak correlations or nonlinear relationships. The included studies generally reported positive associations between AI-based and perceptual intelligibility measures, although the performance metrics and models varied considerably across studies. Therefore, the findings should be interpreted with caution in clinical applications.
- 31 October 2026
- Published by Korean Academy of Audiology in Audiology and Speech Research
- Vol. 22 (4) , 207-223
Open AccessA Literature Review on Artificial Intelligence-Based Speech Intelligibility Assessment in Motor Speech Disorders
This scoping review aimed to examine the methodological characteristics and performance of artificial intelligence (AI)-based speech intelligibility assessment in individuals with acquired motor speech disorders. It also evaluated the clinical applicability of AI-based assessment by reviewing the relationship between AI-based assessments and human perceptual ratings. Four international databases (PubMed, IEEE Xplore, Web of Science, and ScienceDirect) and three Korean databases (KISS, DBPIA, and RISS) were searched up to March 2026. Fifteen studies were selected after applying predefined exclusion criteria. The selected studies were classified and synthesized descriptively without risk-of-bias assessment. 1) Reading tasks were the most frequently used speech task across the 15 included studies, regardless of the data collection method. 2) A wide range of automatic speech recognition systems (e.g., Google Cloud ASR API; Google LLC, Mountain View, CA, USA) and machine learning and deep learning models (e.g., bidirectional long-short term memory, convolutional neural networks, support vector machines, and linear regression) were used, with no single approach predominating. 3) Many studies reported strong correlations or high levels of agreement between AI-based assessments and human perceptual ratings, whereas some reported weak correlations or nonlinear relationships. The included studies generally reported positive associations between AI-based and perceptual intelligibility measures, although the performance metrics and models varied considerably across studies. Therefore, the findings should be interpreted with caution in clinical applications.
- 6 October 2026
- Published by Elsevier in Journal of Voice
Divergent HNR and CPP Aging Trajectories in Korean Connected Speech
- 6 October 2026
- Published by Elsevier in Journal of Voice
Divergent HNR and CPP Aging Trajectories in Korean Connected Speech
- Open Access
- 20 May 2026
- Published by Korean Society of Otorhinolaryngology-Head and Neck Surgery in Korean Journal of Otorhinolaryngology-Head and Neck Surgery
- Vol. 69 (7) , 323-328
Clinical Applicability of Whisper-Based Automatic Transcription for Korean Speech Audiometry Sentence Tests in Older Adults with Hearing Loss
Presbycusis often disrupts sentence-level communication, while clinical speech audiometry still depends on labor-intensive scoring. This exploratory two-case study examined whether the large-scale automatic speech recognition (ASR) model Whisper large-v3 can reliably transcribe sentence recognition performances in elderly listeners with presbycusis and how age-related auditory and speech characteristics shape its error patterns. Two native Korean elderly listeners with symmetric sensorineural hearing loss (S-001: moderate steeply sloping; S-002: mild sloping highfrequency loss) completed the Korean Speech Audiometry (KSA) sentence test (80 sentences). Their repetitions in the quiet were recorded and independently transcribed by four experienced audiologists. Expert transcriptions were compared with Whisper outputs generated under fixed decoding parameters and standardized text normalization, using sentence match rate, word error rate (WER), and character error rate (CER). Whisper showed relatively low CER (about 8%-15%) but substantially higher word- and sentence-level errors (WER 30% for S-001 vs. 18% for S-002; sentence match 38.5% vs. 71.2%). Errors clustered in the high-frequency fricatives/affricates, final consonants, low-frequency and polysyllabic words, and longer syntactically complex sentences. Better clinical speech audiometry scores (KSA sentence/word recognition and word recognition score) were associated with higher ASR sentence match rates and lower WER/CER across the two cases. Generic ASR partially agreed with expert transcriptions, suggesting potential as a complementary tool, but elderly-and hearing loss-tailored ASR models and test designs are needed for reliable AI-based sentence recognition.
- 20 May 2026
- Published by Korean Society of Otorhinolaryngology-Head and Neck Surgery in Korean Journal of Otorhinolaryngology-Head and Neck Surgery
- Vol. 69 (7) , 323-328
Open AccessClinical Applicability of Whisper-Based Automatic Transcription for Korean Speech Audiometry Sentence Tests in Older Adults with Hearing Loss
Presbycusis often disrupts sentence-level communication, while clinical speech audiometry still depends on labor-intensive scoring. This exploratory two-case study examined whether the large-scale automatic speech recognition (ASR) model Whisper large-v3 can reliably transcribe sentence recognition performances in elderly listeners with presbycusis and how age-related auditory and speech characteristics shape its error patterns. Two native Korean elderly listeners with symmetric sensorineural hearing loss (S-001: moderate steeply sloping; S-002: mild sloping highfrequency loss) completed the Korean Speech Audiometry (KSA) sentence test (80 sentences). Their repetitions in the quiet were recorded and independently transcribed by four experienced audiologists. Expert transcriptions were compared with Whisper outputs generated under fixed decoding parameters and standardized text normalization, using sentence match rate, word error rate (WER), and character error rate (CER). Whisper showed relatively low CER (about 8%-15%) but substantially higher word- and sentence-level errors (WER 30% for S-001 vs. 18% for S-002; sentence match 38.5% vs. 71.2%). Errors clustered in the high-frequency fricatives/affricates, final consonants, low-frequency and polysyllabic words, and longer syntactically complex sentences. Better clinical speech audiometry scores (KSA sentence/word recognition and word recognition score) were associated with higher ASR sentence match rates and lower WER/CER across the two cases. Generic ASR partially agreed with expert transcriptions, suggesting potential as a complementary tool, but elderly-and hearing loss-tailored ASR models and test designs are needed for reliable AI-based sentence recognition.
- Open Access
- 28 February 2026
- Published by Elsevier in SSRN Electronic Journal
Robustness of Forced Alignment to ASR Transcription Errors: Large-Scale Evaluation on Korean L2 English Speech
Forced alignment (FA) is essential for large-scale phonetic analysis, yet its robustness to transcript disagreement in spontaneous non-native speech is not well characterized. We evaluate three practical pipelines for Korean L2 English using 30,823 prompt responses (2,096 speakers; ≈660 h) across five proficiency levels: Montreal Forced Aligner (MFA) with the corpus’ released, human-edited transcripts; MFA with disfluency-preserving CrisperWhisper transcripts; and a hybrid pipeline that segments recordings with WhisperX and aligns CrisperWhisper transcripts with MFA. Transcript disagreement between released and ASR transcripts was substantial (WER 42–145% by proficiency). Nevertheless, MFA returned alignments for 99.6% of files in all conditions. Across 10.8M phone tokens, 98.5% of phone categories (65/66) showed negligible cross-pipeline duration differences (|d|<0.2). Disfluency-preserving transcripts increased aligned phone yield by 4.2%, and the hybrid pipeline by 12.3%, while reducing duration-outlier rates (1.62%→1.38%). One consistent exception was /ʌ/ (ARPAbet AH1) (d=−0.30; 38.9 ms), plausibly reflecting L1-driven differences in vowel reduction in Korean-accented English. Finally, we propose an automatic mismatch detector that combines TextGrid word coverage with VAD-based speech activity and independent transcript cues to localize likely transcript–audio inconsistencies for quality control. These results demonstrate MFA’s acoustic robustness and support cautious use of verbatim ASR transcripts for scalable L2 corpus development and downstream speech-technology applications, including automated pronunciation assessment.
- 28 February 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessRobustness of Forced Alignment to ASR Transcription Errors: Large-Scale Evaluation on Korean L2 English Speech
Forced alignment (FA) is essential for large-scale phonetic analysis, yet its robustness to transcript disagreement in spontaneous non-native speech is not well characterized. We evaluate three practical pipelines for Korean L2 English using 30,823 prompt responses (2,096 speakers; ≈660 h) across five proficiency levels: Montreal Forced Aligner (MFA) with the corpus’ released, human-edited transcripts; MFA with disfluency-preserving CrisperWhisper transcripts; and a hybrid pipeline that segments recordings with WhisperX and aligns CrisperWhisper transcripts with MFA. Transcript disagreement between released and ASR transcripts was substantial (WER 42–145% by proficiency). Nevertheless, MFA returned alignments for 99.6% of files in all conditions. Across 10.8M phone tokens, 98.5% of phone categories (65/66) showed negligible cross-pipeline duration differences (|d|<0.2). Disfluency-preserving transcripts increased aligned phone yield by 4.2%, and the hybrid pipeline by 12.3%, while reducing duration-outlier rates (1.62%→1.38%). One consistent exception was /ʌ/ (ARPAbet AH1) (d=−0.30; 38.9 ms), plausibly reflecting L1-driven differences in vowel reduction in Korean-accented English. Finally, we propose an automatic mismatch detector that combines TextGrid word coverage with VAD-based speech activity and independent transcript cues to localize likely transcript–audio inconsistencies for quality control. These results demonstrate MFA’s acoustic robustness and support cautious use of verbatim ASR transcripts for scalable L2 corpus development and downstream speech-technology applications, including automated pronunciation assessment.
- Open Access
- 26 January 2026
- Published by MDPI AG in Behavioral Sciences
- Vol. 16 (2) , 179
Decoding How Articulation and Pauses Influence Pronunciation Proficiency in Korean Learners of English
This study investigates how temporal fluency cues shape human ratings of L2 English pronunciation in Korean learners, using a large read-speech corpus annotated with five-point pronunciation scores. We focus on two timing-derived measures—articulation rate (AR) and mean silence duration (SilMean)—and examine whether these cues predict (i) articulation-accuracy ratings and (ii) prosody/fluency ratings. To account for dependencies in corpus data and to control for key learner- and task-level covariates, we fitted cumulative link mixed models with random intercepts for speakers and scripts, including proficiency band (ability), age, gender, and test type as fixed effects. Across models, faster articulation and shorter silent intervals were associated with higher articulation ratings, and a combined model including both AR and SilMean provided the best fit (lowest AIC). Temporal cues were even more strongly associated with prosody ratings, supporting construct alignment between timing measures and the prosody dimension of the rubric. Marginal predicted probabilities illustrate how the likelihood of receiving high ratings (score ≥ 4) increases with AR across proficiency and linguistic-complexity strata (with SilMean held constant), and how long silent intervals reduce these probabilities when AR is held constant. These findings indicate that temporal organization provides robust information about perceived pronunciation quality in read L2 speech and underscore the importance of construct-aware modeling when developing AI-based scoring and feedback systems trained on human-labeled data.
- 26 January 2026
- Published by MDPI AG in Behavioral Sciences
- Vol. 16 (2) , 179
Open AccessDecoding How Articulation and Pauses Influence Pronunciation Proficiency in Korean Learners of English
This study investigates how temporal fluency cues shape human ratings of L2 English pronunciation in Korean learners, using a large read-speech corpus annotated with five-point pronunciation scores. We focus on two timing-derived measures—articulation rate (AR) and mean silence duration (SilMean)—and examine whether these cues predict (i) articulation-accuracy ratings and (ii) prosody/fluency ratings. To account for dependencies in corpus data and to control for key learner- and task-level covariates, we fitted cumulative link mixed models with random intercepts for speakers and scripts, including proficiency band (ability), age, gender, and test type as fixed effects. Across models, faster articulation and shorter silent intervals were associated with higher articulation ratings, and a combined model including both AR and SilMean provided the best fit (lowest AIC). Temporal cues were even more strongly associated with prosody ratings, supporting construct alignment between timing measures and the prosody dimension of the rubric. Marginal predicted probabilities illustrate how the likelihood of receiving high ratings (score ≥ 4) increases with AR across proficiency and linguistic-complexity strata (with SilMean held constant), and how long silent intervals reduce these probabilities when AR is held constant. These findings indicate that temporal organization provides robust information about perceived pronunciation quality in read L2 speech and underscore the importance of construct-aware modeling when developing AI-based scoring and feedback systems trained on human-labeled data.
- Open Access
- 20 January 2026
- Published by The Korean Audiological Society in Journal of Audiology and Otology
- Vol. 30 (1) , 23-33
Efficacy of Self-Directed Auditory Rehabilitation in Individuals With Hearing Loss: A Systematic Review and Meta-Analysis
Background and Objectives: Hearing loss negatively affects communication and quality of life. While clinician-led rehabilitation is beneficial, access barriers and poor long-term adherence limit its overall impact. This systematic review evaluated the efficacy of self-directed auditory rehabilitation across the lifespan.Materials and Methods: A systematic search was conducted using five electronic databases for peer-reviewed studies involving children, adults, and older adults with at least mild hearing loss, who participated in self-directed auditory rehabilitation and reported behavioral, objective, or patient-reported auditory outcomes.Results: Twenty-two studies with 1,851 screened records were included in the synthesis, which encompassed varied populations, devices, and rehabilitation formats. Random-effects meta-analyses showed significant speech recognition improvement in pediatric participants (Cohen’s d=1.32, standard error [SE]=0.12, 95% confidence interval [CI]: 1.08-1.56; p<0.05), with effects maintained at 1-3 months. A moderately significant benefit was observed in both adults and older adults (Cohen’s d=0.55, SE=0.05, 95% CI: 0.45-0.65). Although publication bias was present in some adult studies, the sensitivity analyses confirmed the robustness of the results.Conclusions: Self-directed auditory rehabilitation produces meaningful and sustained speech recognition benefits in individuals with hearing loss, endorsing its integration as an accessible adjunct to conventional audiologic care. Future studies should refine the program content and reduce intervention variability.
- 20 January 2026
- Published by The Korean Audiological Society in Journal of Audiology and Otology
- Vol. 30 (1) , 23-33
Open AccessEfficacy of Self-Directed Auditory Rehabilitation in Individuals With Hearing Loss: A Systematic Review and Meta-Analysis
Background and Objectives: Hearing loss negatively affects communication and quality of life. While clinician-led rehabilitation is beneficial, access barriers and poor long-term adherence limit its overall impact. This systematic review evaluated the efficacy of self-directed auditory rehabilitation across the lifespan.Materials and Methods: A systematic search was conducted using five electronic databases for peer-reviewed studies involving children, adults, and older adults with at least mild hearing loss, who participated in self-directed auditory rehabilitation and reported behavioral, objective, or patient-reported auditory outcomes.Results: Twenty-two studies with 1,851 screened records were included in the synthesis, which encompassed varied populations, devices, and rehabilitation formats. Random-effects meta-analyses showed significant speech recognition improvement in pediatric participants (Cohen’s d=1.32, standard error [SE]=0.12, 95% confidence interval [CI]: 1.08-1.56; p<0.05), with effects maintained at 1-3 months. A moderately significant benefit was observed in both adults and older adults (Cohen’s d=0.55, SE=0.05, 95% CI: 0.45-0.65). Although publication bias was present in some adult studies, the sensitivity analyses confirmed the robustness of the results.Conclusions: Self-directed auditory rehabilitation produces meaningful and sustained speech recognition benefits in individuals with hearing loss, endorsing its integration as an accessible adjunct to conventional audiologic care. Future studies should refine the program content and reduce intervention variability.
- Open Access
- 28 December 2025
- Published by Korean Academy of Speech-Language Pathology and Audiology in Communication Sciences & Disorders
- Vol. 30 (4) , 741-750
Longitudinal Investigation of Early Vowel Development in Korean Infants
Objectives: This study investigated early vowel development in Korean-learning infants by analyzing longitudinal vocalizations at 6-8 and 12-14 months. The study aimed to identify developmental changes in the frequency and distribution of vowels and diphthongs, providing empirical data on the early organization of the Korean vowel system. Methods: Fourteen typically developing infants (7 boys, 7 girls) participated. Naturalistic daylong recordings were obtained in the home environment using the Language Environment Analysis (LENA) system. From each session, 100 minutes with the highest infant vocal activity were extracted and analyzed with the AACT software. All utterances were segmented into breath groups and phonetically transcribed by trained coders in blind conditions. Vowel productions were analyzed based on the frequencies of isolated and consonant vowel (CV)-bound vowels and diphthongs within canonical babbling. Longitudinal changes were examined between 6-8 and 12-14 months. Results: The total number of utterances and syllables increased significantly across the two age periods. Across both ages, /a/, /ɯ/, and /ε/ were most frequently produced, with /a/ consistently dominant. High back vowels (/o/, /u/) remained rare, while diphthongs such as /ja/ and /wa/ were produced with relatively high frequency at 12-14 months, suggesting increasing articulatory coordination. Conclusion: By longitudinally analyzing vowel productions in the babbling of Korean infants aged 6 to 14 months, this study delineates the developmental trajectory of the Korean vowel system and provides baseline data for understanding early phonological development.
- 28 December 2025
- Published by Korean Academy of Speech-Language Pathology and Audiology in Communication Sciences & Disorders
- Vol. 30 (4) , 741-750
Open AccessLongitudinal Investigation of Early Vowel Development in Korean Infants
Objectives: This study investigated early vowel development in Korean-learning infants by analyzing longitudinal vocalizations at 6-8 and 12-14 months. The study aimed to identify developmental changes in the frequency and distribution of vowels and diphthongs, providing empirical data on the early organization of the Korean vowel system. Methods: Fourteen typically developing infants (7 boys, 7 girls) participated. Naturalistic daylong recordings were obtained in the home environment using the Language Environment Analysis (LENA) system. From each session, 100 minutes with the highest infant vocal activity were extracted and analyzed with the AACT software. All utterances were segmented into breath groups and phonetically transcribed by trained coders in blind conditions. Vowel productions were analyzed based on the frequencies of isolated and consonant vowel (CV)-bound vowels and diphthongs within canonical babbling. Longitudinal changes were examined between 6-8 and 12-14 months. Results: The total number of utterances and syllables increased significantly across the two age periods. Across both ages, /a/, /ɯ/, and /ε/ were most frequently produced, with /a/ consistently dominant. High back vowels (/o/, /u/) remained rare, while diphthongs such as /ja/ and /wa/ were produced with relatively high frequency at 12-14 months, suggesting increasing articulatory coordination. Conclusion: By longitudinally analyzing vowel productions in the babbling of Korean infants aged 6 to 14 months, this study delineates the developmental trajectory of the Korean vowel system and provides baseline data for understanding early phonological development.
- Open Access
- 26 November 2024
- Published by MDPI AG in Applied Sciences
- Vol. 14 (23) , 10972
How Much Does the Dynamic F0 Curve Affect the Expression of Emotion in Utterances?
The modulation of vocal elements, such as pitch, loudness, and duration, plays a crucial role in conveying both linguistic information and the speaker’s emotional state. While acoustic features like fundamental frequency (F0) variability have been widely studied in emotional speech analysis, accurately classifying emotion remains challenging due to the complex and dynamic nature of vocal expressions. Traditional analytical methods often oversimplify these dynamics, potentially overlooking intricate patterns indicative of specific emotions. This study examines the influences of emotion and temporal variation on dynamic F0 contours in the analytical framework, utilizing a dataset valuable for its diverse emotional expressions. However, the analysis is constrained by the limited variety of sentences employed, which may affect the generalizability of the findings to broader linguistic contexts. We utilized the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), focusing on eight distinct emotional states performed by 24 professional actors. Sonorant segments were extracted, and F0 measurements were converted into semitones relative to a 100 Hz baseline to standardize pitch variations. By employing Generalized Additive Mixed Models (GAMMs), we modeled non-linear trajectories of F0 contours over time, accounting for fixed effects (emotions) and random effects (individual speaker variability). Our analysis revealed that incorporating emotion-specific, non-linear time effects and individual speaker differences significantly improved the model’s explanatory power, ultimately explaining up to 66.5% of the variance in the F0. The inclusion of random smooths for time within speakers captured individual temporal modulation patterns, providing a more accurate representation of emotional speech dynamics. The results demonstrate that dynamic modeling of F0 contours using GAMMs enhances the accuracy of emotion classification in speech. This approach captures the nuanced pitch patterns associated with different emotions and accounts for individual variability among speakers. The findings contribute to a deeper understanding of the vocal expression of emotions and offer valuable insights for advancing speech emotion recognition systems.
- 26 November 2024
- Published by MDPI AG in Applied Sciences
- Vol. 14 (23) , 10972
Open AccessHow Much Does the Dynamic F0 Curve Affect the Expression of Emotion in Utterances?
The modulation of vocal elements, such as pitch, loudness, and duration, plays a crucial role in conveying both linguistic information and the speaker’s emotional state. While acoustic features like fundamental frequency (F0) variability have been widely studied in emotional speech analysis, accurately classifying emotion remains challenging due to the complex and dynamic nature of vocal expressions. Traditional analytical methods often oversimplify these dynamics, potentially overlooking intricate patterns indicative of specific emotions. This study examines the influences of emotion and temporal variation on dynamic F0 contours in the analytical framework, utilizing a dataset valuable for its diverse emotional expressions. However, the analysis is constrained by the limited variety of sentences employed, which may affect the generalizability of the findings to broader linguistic contexts. We utilized the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), focusing on eight distinct emotional states performed by 24 professional actors. Sonorant segments were extracted, and F0 measurements were converted into semitones relative to a 100 Hz baseline to standardize pitch variations. By employing Generalized Additive Mixed Models (GAMMs), we modeled non-linear trajectories of F0 contours over time, accounting for fixed effects (emotions) and random effects (individual speaker variability). Our analysis revealed that incorporating emotion-specific, non-linear time effects and individual speaker differences significantly improved the model’s explanatory power, ultimately explaining up to 66.5% of the variance in the F0. The inclusion of random smooths for time within speakers captured individual temporal modulation patterns, providing a more accurate representation of emotional speech dynamics. The results demonstrate that dynamic modeling of F0 contours using GAMMs enhances the accuracy of emotion classification in speech. This approach captures the nuanced pitch patterns associated with different emotions and accounts for individual variability among speakers. The findings contribute to a deeper understanding of the vocal expression of emotions and offer valuable insights for advancing speech emotion recognition systems.
- Open Access
- 1 October 2024
- Published by MDPI AG in Preprints.org
How Much Does the Dynamic F0 Curve Affect the Expression of Emotion in Utterances?
The modulation of vocal elements such as pitch, loudness, and duration plays a crucial role in conveying both linguistic information and the speaker’s emotional state. While acoustic features like fundamental frequency (F0) variability have been widely studied in emotional speech analysis, challenges remain in accurately classifying emotions due to the complex and dynamic nature of vocal expressions. Traditional analytical methods often oversimplify these dynamics, potentially overlooking intricate patterns indicative of specific emotions. This study aims to enhance emotion classification in speech by directly incorporating dynamic F0 contours into the analytical framework using Generalized Additive Mixed Models (GAMMs). We utilized the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), focusing on eight distinct emotional states expressed by 24 professional actors. Sonorant segments were extracted, and F0 measurements were converted into semitones relative to a 100 Hz baseline to standardize pitch variations. By employing GAMMs, we modeled non-linear trajectories of F0 contours over time, accounting for both fixed effects (emotions) and random effects (individual speaker variability). Our analysis revealed that incorporating emotion-specific non-linear time effects and individual speaker differences significantly improved the model’s explanatory power, ultimately explaining up to 66.5% of the variance in F0. The inclusion of random smooths for time within speakers captured individual temporal modulation patterns, providing a more accurate representation of emotional speech dynamics. The results demonstrate that dynamic modeling of F0 contours using GAMMs enhances the accuracy of emotion classification in speech. This approach captures the nuanced pitch patterns associated with different emotions and accounts for individual variability among speakers. The findings contribute to a deeper understanding of the vocal expression of emotions and offer valuable insights for advancing speech emotion recognition systems.
- 1 October 2024
- Published by MDPI AG in Preprints.org
Open AccessHow Much Does the Dynamic F0 Curve Affect the Expression of Emotion in Utterances?
The modulation of vocal elements such as pitch, loudness, and duration plays a crucial role in conveying both linguistic information and the speaker’s emotional state. While acoustic features like fundamental frequency (F0) variability have been widely studied in emotional speech analysis, challenges remain in accurately classifying emotions due to the complex and dynamic nature of vocal expressions. Traditional analytical methods often oversimplify these dynamics, potentially overlooking intricate patterns indicative of specific emotions. This study aims to enhance emotion classification in speech by directly incorporating dynamic F0 contours into the analytical framework using Generalized Additive Mixed Models (GAMMs). We utilized the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), focusing on eight distinct emotional states expressed by 24 professional actors. Sonorant segments were extracted, and F0 measurements were converted into semitones relative to a 100 Hz baseline to standardize pitch variations. By employing GAMMs, we modeled non-linear trajectories of F0 contours over time, accounting for both fixed effects (emotions) and random effects (individual speaker variability). Our analysis revealed that incorporating emotion-specific non-linear time effects and individual speaker differences significantly improved the model’s explanatory power, ultimately explaining up to 66.5% of the variance in F0. The inclusion of random smooths for time within speakers captured individual temporal modulation patterns, providing a more accurate representation of emotional speech dynamics. The results demonstrate that dynamic modeling of F0 contours using GAMMs enhances the accuracy of emotion classification in speech. This approach captures the nuanced pitch patterns associated with different emotions and accounts for individual variability among speakers. The findings contribute to a deeper understanding of the vocal expression of emotions and offer valuable insights for advancing speech emotion recognition systems.
- Open Access
- 30 September 2024
- Published by The Korean Society of Speech Sciences in Phonetics and Speech Sciences
- Vol. 16 (3) , 87-94
Automatic detection of speech sound disorder in children using automatic speech recognition and audio classification*
말소리와 음성과학(Phonetics and Speech Sciences)
- 30 September 2024
- Published by The Korean Society of Speech Sciences in Phonetics and Speech Sciences
- Vol. 16 (3) , 87-94
Open AccessAutomatic detection of speech sound disorder in children using automatic speech recognition and audio classification*
말소리와 음성과학(Phonetics and Speech Sciences)