Karen Rosero
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Karen Rosero has a total of 23 co-authors. Here are the top co-authors by number of shared publications:
- Bruno Masiero
Faculdade de Engenharia Elétrica e de Computação Universidade Estadual de Campinas Campinas Brazil
4 shared publications - Felipe Grijalva
Colegio de Ciencias e Ingenierías “El Politécnico”, Universidad San Francisco de Quito USFQ, Diego de Robles s/n, Quito 170157, Ecuador
4 shared publications - Rami R. Hallac
University of Texas Southwestern Medical Center
4 shared publications - Carlos Busso
Language Technologies Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
3 shared publications - Lucas M. Harrison
Department of Plastic Surgery, University of Texas Southwestern Medical Center
2 shared publications - Nathaly Orozco Garzón
ETEL Research Group, Faculty of Engineering and Applied Sciences, Networking and Telecommunications Engineering, Universidad de Las Américas (UDLA), Quito 170503, Ecuador
2 shared publications
Latest Publications
- Open Access
- 4 October 2026
- Published by Association for Computing Machinery (ACM)
- p. 546-555
Multimodal Biomarkers of Dysarthria: Severity-Conditioned Contrastive Alignment of Speech, Text, and Facial Asymmetry
Dysarthria is a motor speech disorder that affects speech production, and severity assessment can benefit from combining acoustic and visual biomarkers. While recent advances have improved unimodal-based evaluations, most systems do not fully leverage facial motion, asymmetry, or the relationship between the produced speech and its intended linguistic content. In this work, we propose a framework for MultiModal Dysarthria Severity assessment (MMDyS), which exploits the alignment of speech, text, and visual biomarkers. We introduce a severity-conditioned contrastive objective that treats speech–text mismatch as a continuous signal of impairment, while the visual modality relies on hemiface lateralization to capture asymmetric articulatory patterns, complemented by temporal segmentation and flow motion descriptors. These representations are integrated through a mixture-of-experts mechanism with cross-modal alignment, enabling adaptive weighting of speech, text, and visual evidence. Evaluated on the Mandarin speech dysarthria multimodal (MSDM) benchmark, MMDyS achieves a subject-level macro F1 of 0.92 and the composite multimodal dysarthria severity assessment (MDSA) score of 11.7% over the current state-of-the-art. Our results show that modeling speech–text divergence and facial asymmetry leads to more informative and robust multimodal representations for dysarthria severity assessment. Code is available at: https://github.com/Lab-MSP/MMDyS.
- 4 October 2026
- Published by Association for Computing Machinery (ACM)
- p. 546-555
Open AccessMultimodal Biomarkers of Dysarthria: Severity-Conditioned Contrastive Alignment of Speech, Text, and Facial Asymmetry
Dysarthria is a motor speech disorder that affects speech production, and severity assessment can benefit from combining acoustic and visual biomarkers. While recent advances have improved unimodal-based evaluations, most systems do not fully leverage facial motion, asymmetry, or the relationship between the produced speech and its intended linguistic content. In this work, we propose a framework for MultiModal Dysarthria Severity assessment (MMDyS), which exploits the alignment of speech, text, and visual biomarkers. We introduce a severity-conditioned contrastive objective that treats speech–text mismatch as a continuous signal of impairment, while the visual modality relies on hemiface lateralization to capture asymmetric articulatory patterns, complemented by temporal segmentation and flow motion descriptors. These representations are integrated through a mixture-of-experts mechanism with cross-modal alignment, enabling adaptive weighting of speech, text, and visual evidence. Evaluated on the Mandarin speech dysarthria multimodal (MSDM) benchmark, MMDyS achieves a subject-level macro F1 of 0.92 and the composite multimodal dysarthria severity assessment (MDSA) score of 11.7% over the current state-of-the-art. Our results show that modeling speech–text divergence and facial asymmetry leads to more informative and robust multimodal representations for dysarthria severity assessment. Code is available at: https://github.com/Lab-MSP/MMDyS.
- Open Access
- 12 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 506-515
DiffusionCleft: Facial Anomaly Synthesis Guided by Text
- 12 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 506-515
Open AccessDiffusionCleft: Facial Anomaly Synthesis Guided by Text
- Open Access
- 22 January 2025
- Published by SAGE Publications in The Cleft Palate Craniofacial Journal
- Vol. 62 (2) , 289-299
Deep Learning-Based Assessment of Lip Symmetry for Patients With Repaired Cleft Lip
Objective: Post-surgical lip symmetry assessment is a key indicator of cleft repair success. Traditional methods rely on distances between anatomical landmarks, which are impractical for video analysis and overlook texture and appearance. We propose an artificial intelligence (AI) approach to automate this process, analyzing lateral lip morphology for a quantitative symmetry evaluation. Design: We utilize contrastive learning to quantify lip symmetry by measuring the similarity between the representations of the sides, which is subsequently used to classify the severity of asymmetry. Our model does not require patient images for training. Instead, we introduce dissimilarities in face images from open datasets using two methods: temporal misalignment for video frames and face transformations to simulate lip asymmetry observed in the target population. The model differentiates the left and right image representations to assess asymmetry. We evaluated our model on 146 images of patients with repaired cleft lip. Results: The deep learning model trained with face transformations categorized patient images into five asymmetry levels, achieving a weighted accuracy of 75% and a Pearson correlation of 0.31 with medical expert human evaluations. The model utilizing temporal misalignment achieved a weighted accuracy of 69% and a Pearson correlation of 0.27 for the same classification task. Conclusions: We propose an automated approach for assessing lip asymmetry in patients with repaired cleft lip by transforming facial images of control subjects to train a deep learning model, eliminating manual anatomical landmarks. Our promising results provide a more efficient and objective tool for evaluating surgical outcomes.
- 22 January 2025
- Published by SAGE Publications in The Cleft Palate Craniofacial Journal
- Vol. 62 (2) , 289-299
Open AccessDeep Learning-Based Assessment of Lip Symmetry for Patients With Repaired Cleft Lip
Objective: Post-surgical lip symmetry assessment is a key indicator of cleft repair success. Traditional methods rely on distances between anatomical landmarks, which are impractical for video analysis and overlook texture and appearance. We propose an artificial intelligence (AI) approach to automate this process, analyzing lateral lip morphology for a quantitative symmetry evaluation. Design: We utilize contrastive learning to quantify lip symmetry by measuring the similarity between the representations of the sides, which is subsequently used to classify the severity of asymmetry. Our model does not require patient images for training. Instead, we introduce dissimilarities in face images from open datasets using two methods: temporal misalignment for video frames and face transformations to simulate lip asymmetry observed in the target population. The model differentiates the left and right image representations to assess asymmetry. We evaluated our model on 146 images of patients with repaired cleft lip. Results: The deep learning model trained with face transformations categorized patient images into five asymmetry levels, achieving a weighted accuracy of 75% and a Pearson correlation of 0.31 with medical expert human evaluations. The model utilizing temporal misalignment achieved a weighted accuracy of 69% and a Pearson correlation of 0.27 for the same classification task. Conclusions: We propose an automated approach for assessing lip asymmetry in patients with repaired cleft lip by transforming facial images of control subjects to train a deep learning model, eliminating manual anatomical landmarks. Our promising results provide a more efficient and objective tool for evaluating surgical outcomes.
- Open Access
- 2 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 181553-181569
w2v-SELD: A Sound Event Localization and Detection Framework for Self-Supervised Spatial Audio Pre-Training
Sound Event Localization and Detection (SELD) is a critical challenge in various industrial applications, such as autonomous systems, smart cities, and audio surveillance, which require accurate identification and localization of sound events in complex environments. Traditional supervised approaches heavily rely on large, annotated multichannel audio datasets, which are expensive and time-consuming to produce. This paper addresses this limitation by introducing the w2v-SELD architecture, a self-supervised model adapted from the wav2vec 2.0 framework to learn effective sound event representations directly from raw, unlabeled 3D audio data. The proposed model follows a two-stage process: pre-training on large, unlabeled 3D audio datasets to capture high-level features, followed by fine-tuning with a smaller, labeled SELD dataset. Experimental results show that our w2v-SELD method outperforms baseline models on Detection and Classification of Acoustic Scenes and Events (DCASE) challenges, achieving a 66% improvement for DCASE TAU-2019 and a 57% improvement on DCASE TAU-2020 with respect to baseline systems. The w2v-SELD model performs competitively with state-of-the-art supervised methods, highlighting its potential to significantly reduce the dependency on labeled data in industrial SELD applications. The code and pre-trained parameters of our w2v-SELD model are available in this repository.
- 2 December 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 181553-181569
Open Accessw2v-SELD: A Sound Event Localization and Detection Framework for Self-Supervised Spatial Audio Pre-Training
Sound Event Localization and Detection (SELD) is a critical challenge in various industrial applications, such as autonomous systems, smart cities, and audio surveillance, which require accurate identification and localization of sound events in complex environments. Traditional supervised approaches heavily rely on large, annotated multichannel audio datasets, which are expensive and time-consuming to produce. This paper addresses this limitation by introducing the w2v-SELD architecture, a self-supervised model adapted from the wav2vec 2.0 framework to learn effective sound event representations directly from raw, unlabeled 3D audio data. The proposed model follows a two-stage process: pre-training on large, unlabeled 3D audio datasets to capture high-level features, followed by fine-tuning with a smaller, labeled SELD dataset. Experimental results show that our w2v-SELD method outperforms baseline models on Detection and Classification of Acoustic Scenes and Events (DCASE) challenges, achieving a 66% improvement for DCASE TAU-2019 and a 57% improvement on DCASE TAU-2020 with respect to baseline systems. The w2v-SELD model performs competitively with state-of-the-art supervised methods, highlighting its potential to significantly reduce the dependency on labeled data in industrial SELD applications. The code and pre-trained parameters of our w2v-SELD model are available in this repository.
- Open Access
- 4 November 2024
- Published by Association for Computing Machinery (ACM)
- p. 479-487
Lip Abnormality Detection for Patients with Repaired Cleft Lip and Palate: A Lip Normalization Approach
The cleft lip condition arises from the incomplete fusion of oral and labial structures during fetal development, impacting vital functions. After surgical closure, patients commonly present with abnormal lip shape, which may require secondary revision surgery for both aesthetic and functional improvement. However, a lack of standardized evaluation methods complicates decision-making for secondary surgery. To address this limitation, we propose a transformer-based lip normalization approach that filters out abnormalities and achieves a standardized appearance while preserving individual anatomy. An innovation of our approach is a lip transformation method using available face datasets to mimic repaired cleft lip shapes, enabling the training of deep learning models without using patients’ data. We employ a Siamese convolutional neural network that processes pre- and post-normalization images to detect lip abnormalities with an accuracy of 89%. We compare our approach with a single-branch model without lip normalization, which reached an accuracy of 60%. Our approach has the potential to provide an impartial view to determine the need for revision surgery while also assisting in the selection of healthcare tools specialized for patients with repaired cleft lip. The code for this work is available in our anonymous repository 1.
- 4 November 2024
- Published by Association for Computing Machinery (ACM)
- p. 479-487
Open AccessLip Abnormality Detection for Patients with Repaired Cleft Lip and Palate: A Lip Normalization Approach
The cleft lip condition arises from the incomplete fusion of oral and labial structures during fetal development, impacting vital functions. After surgical closure, patients commonly present with abnormal lip shape, which may require secondary revision surgery for both aesthetic and functional improvement. However, a lack of standardized evaluation methods complicates decision-making for secondary surgery. To address this limitation, we propose a transformer-based lip normalization approach that filters out abnormalities and achieves a standardized appearance while preserving individual anatomy. An innovation of our approach is a lip transformation method using available face datasets to mimic repaired cleft lip shapes, enabling the training of deep learning models without using patients’ data. We employ a Siamese convolutional neural network that processes pre- and post-normalization images to detect lip abnormalities with an accuracy of 89%. We compare our approach with a single-branch model without lip normalization, which reached an accuracy of 60%. Our approach has the potential to provide an impartial view to determine the need for revision surgery while also assisting in the selection of healthcare tools specialized for patients with repaired cleft lip. The code for this work is available in our anonymous repository 1.
- Open Access
- 12 December 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 141289-141298
A-MAD: An Adaptive-Microseismic Activity Detector Based on Gaussian Mixture Models and Spectral Subtraction
Continuous monitoring of active volcanoes is essential for understanding their behavior and providing accurate forecasts and warnings. Automating the detection of volcano microseismic events plays a crucial role in large-scale analysis, early warning systems, and efficient handling of large-scale data. This paper presents a novel approach for the automated detection of microseismic events associated with volcanic activity, focusing on the Cotopaxi volcano in Ecuador. Inspired by voice activity detection (VAD) systems used in speech processing, we propose an Adaptive-Microseismic Activity Detector (A-MAD) that modifies VAD techniques to identify frames within seismic signals containing volcanic activity. The A-MAD system incorporates a spectral subtraction stage to mitigate the impact of environmental noise and employs Gaussian Mixture Models (GMMs) to model the probabilistic distribution of microseismic events. Mel Frequency Cepstral Coefficients (MFCCs) are used as features to describe the seismic signals, enabling adaptability for varying noise levels. Experimental results on signals from Cotopaxi Volcano in Ecuador demonstrate the effectiveness of the A-MAD system, achieving high accuracy (96.39% for discrete events and 98.45% for continuous signals) while meeting the efficiency requirements of volcano monitoring institutions. This work contributes to the advancement of early warning systems for volcanic eruptions, providing a robust and adaptive approach for microseismic activity detection.
- 12 December 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 141289-141298
Open AccessA-MAD: An Adaptive-Microseismic Activity Detector Based on Gaussian Mixture Models and Spectral Subtraction
Continuous monitoring of active volcanoes is essential for understanding their behavior and providing accurate forecasts and warnings. Automating the detection of volcano microseismic events plays a crucial role in large-scale analysis, early warning systems, and efficient handling of large-scale data. This paper presents a novel approach for the automated detection of microseismic events associated with volcanic activity, focusing on the Cotopaxi volcano in Ecuador. Inspired by voice activity detection (VAD) systems used in speech processing, we propose an Adaptive-Microseismic Activity Detector (A-MAD) that modifies VAD techniques to identify frames within seismic signals containing volcanic activity. The A-MAD system incorporates a spectral subtraction stage to mitigate the impact of environmental noise and employs Gaussian Mixture Models (GMMs) to model the probabilistic distribution of microseismic events. Mel Frequency Cepstral Coefficients (MFCCs) are used as features to describe the seismic signals, enabling adaptability for varying noise levels. Experimental results on signals from Cotopaxi Volcano in Ecuador demonstrate the effectiveness of the A-MAD system, achieving high accuracy (96.39% for discrete events and 98.45% for continuous signals) while meeting the efficiency requirements of volcano monitoring institutions. This work contributes to the advancement of early warning systems for volcanic eruptions, providing a robust and adaptive approach for microseismic activity detection.
- 9 June 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 31 (23299290) , 2314-2324
Sound Events Localization and Detection Using Bio-Inspired Gammatone Filters and Temporal Convolutional Neural Networks
The auditory brain circuits are biologically constructed to recognize and localize sounds by encoding a combination of cues that help individuals interpret sounds. The development of computational methods inspired by human capacities has established opportunities for improving machine hearing. Recent studies based on deep learning show that using convolutional recurrent neural networks (CRNNs) is a promising approach for sound event detection and localization in spatial sound. Nevertheless, depending on the sound environment, the performance of these systems is still far from reaching perfect metrics. Therefore, this work intends to boost the performance of state-of-the-art (SOTA) systems by using bio-inspired gammatone auditory filters and intensity vectors (IVs) for the acoustic feature extraction stage, along with the implementation of a temporal convolutional network (TCN) block into a CRNN model, to capture long term dependencies. Three data augmentation techniques are applied to increase the small number of samples in spatial audio datasets. The mentioned stages constitute our proposed Gammatone-based Sound Events Localization and Detection (G-SELD) system, which exceeded the SOTA results on four spatial audio datasets with different levels of acoustical complexity and with up to three sound sources overlapping in time.
- 9 June 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 31 (23299290) , 2314-2324
Sound Events Localization and Detection Using Bio-Inspired Gammatone Filters and Temporal Convolutional Neural Networks
The auditory brain circuits are biologically constructed to recognize and localize sounds by encoding a combination of cues that help individuals interpret sounds. The development of computational methods inspired by human capacities has established opportunities for improving machine hearing. Recent studies based on deep learning show that using convolutional recurrent neural networks (CRNNs) is a promising approach for sound event detection and localization in spatial sound. Nevertheless, depending on the sound environment, the performance of these systems is still far from reaching perfect metrics. Therefore, this work intends to boost the performance of state-of-the-art (SOTA) systems by using bio-inspired gammatone auditory filters and intensity vectors (IVs) for the acoustic feature extraction stage, along with the implementation of a temporal convolutional network (TCN) block into a CRNN model, to capture long term dependencies. Three data augmentation techniques are applied to increase the small number of samples in spatial audio datasets. The mentioned stages constitute our proposed Gammatone-based Sound Events Localization and Detection (G-SELD) system, which exceeded the SOTA results on four spatial audio datasets with different levels of acoustical complexity and with up to three sound sources overlapping in time.
- Open Access
- 30 December 2022
- Published by Sociedade Brasileira de Computação in Revista Eletrônica de Iniciação Científica em Computação
- Vol. 20 (4) , 55-62
Song Emotion Recognition: a Performance Comparison Between Audio Features and Artificial Neural Networks
When songs are composed or performed, there is often an intent by the singer/songwriter of expressing feelings or emotions through it. For humans, matching the emotiveness in a musical composition or performance with the subjective perceptiveness of an audience can be quite challenging. Fortunately, the machine learning approach for this problem is simpler. Usually, it takes a data-set, from which audio features are extracted to present this information to a data-driven model, that will, in turn, train predicting the highest probability of an input song matching a target emotion. In this paper†, we studied the most common features and models used in recent publications to tackle this problem, revealing which ones are best suited for songs a cappella.
- 30 December 2022
- Published by Sociedade Brasileira de Computação in Revista Eletrônica de Iniciação Científica em Computação
- Vol. 20 (4) , 55-62
Open AccessSong Emotion Recognition: a Performance Comparison Between Audio Features and Artificial Neural Networks
When songs are composed or performed, there is often an intent by the singer/songwriter of expressing feelings or emotions through it. For humans, matching the emotiveness in a musical composition or performance with the subjective perceptiveness of an audience can be quite challenging. Fortunately, the machine learning approach for this problem is simpler. Usually, it takes a data-set, from which audio features are extracted to present this information to a data-driven model, that will, in turn, train predicting the highest probability of an input song matching a target emotion. In this paper†, we studied the most common features and models used in recent publications to tackle this problem, revealing which ones are best suited for songs a cappella.
- Open Access
- 27 September 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in TechRxiv
Sound Events Localization and Detection Using Bio-inspired Gammatone Filters and Temporal Convolutional Neural Networks
This manuscript addresses the problem of detecting, classifying, and localizing sound sources in an acoustic scene of spatial audio. We propose using bio-inspired Gammatone auditory filters for the acoustic feature extraction stage and a novel deep learning architecture encompassing convolutional, recurrent, and temporal convolutional blocks. Our system exceeded the state-of-the-art metrics on four spatial audio datasets with different levels of acoustical complexity and up to three sound sources overlapping in time. Furthermore, we also performed a comparative analysis of the gap between machine and human hearing, evidencing that our results have already exceeded the human performance in non-reverberant scenarios.
- 27 September 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in TechRxiv
Open AccessSound Events Localization and Detection Using Bio-inspired Gammatone Filters and Temporal Convolutional Neural Networks
This manuscript addresses the problem of detecting, classifying, and localizing sound sources in an acoustic scene of spatial audio. We propose using bio-inspired Gammatone auditory filters for the acoustic feature extraction stage and a novel deep learning architecture encompassing convolutional, recurrent, and temporal convolutional blocks. Our system exceeded the state-of-the-art metrics on four spatial audio datasets with different levels of acoustical complexity and up to three sound sources overlapping in time. Furthermore, we also performed a comparative analysis of the gap between machine and human hearing, evidencing that our results have already exceeded the human performance in non-reverberant scenarios.