Shakeel Ahmad Sheikh

Apprentissage profond pour la détection du bégaiement: Le bégaiement est un trouble de la parole qui est le plus fréquemment observé parmi les troubles de la parole et se traduit par des comportements principaux. La fastidieuse tâche de détection et d'analyse des schémas de parole des Persons who stutter (PWS), dans le but de les rectifier, est souvent traitée manuellement par les orthophonistes et est biaisée par leurs croyances subjectives. De plus, les systèmes de reconnaissance automatique de la parole, «Automatic speech recognition (ASR)», ne parviennent pas non plus à reconnaître les bégaiements. Cela empêche les personnes handicapées d'accéder à des assistants numériques virtuels tels que Siri, Alexa, etc. Cette thèse tente de développer des systèmes de détection du bégaiement, Stuttering detection (SD), basés sur l'audio qui réussissent à capturer les différentes variabilités des énoncés de bégaiement telles que les styles d'expression, l'âge, les accents, etc., et apprennent des représentations robustes du bégaiement dans le but de fournir une évaluation juste, cohérente et impartiale de la parole bégayée. Alors que la plupart des systèmes SD existants utilisent plusieurs classificateurs binaires pour chaque type de bégaiement, nous présentons un système multi-classe unifié nommé StutterNet capable de détecter plusieurs types de bègues. En abordant le problème du déséquilibre des classes dans le domaine du bégaiement, nous avons étudié l'impact de l'application d'une fonction de perte pondérée et nous avons également présenté un StutterNet multi-contextuel pour améliorer la détection des types minoritaires. En exploitant les informations sur le locuteur et en supposant que les modèles de bégaiement sont invariants par rapport aux méta-données telles que les informations sur le locuteur, nous présentons un apprentissage multi-tâches «Multi-task learning (MTL)» SD qui apprend des représentations robustes discriminant le bégaiement et les invariantes par rapport au locuteur. En raison de la rareté des données non étiquetées, la tâche automatisée de détection du bégaiement est limitée dans son utilisation des modèles d'apprentissages profonds de grande taille lorsqu'il s'agit de capturer différentes variabilités. Nous avons introduit la toute première architecture d'apprentissage auto-supervisé «Self-supervised learning (SSL)», dans le domaine de la détection du bégaiement. L'architecture SSL entraîne d'abord un extracteur de caractéristiques pour une tâche de pré-texte en utilisant une grande quantité de données audio non étiquetées et non bégayantes pour capturer ces différentes variabilités. Puis elle applique l'extracteur de caractéristiques appris à une tâche SD en aval, en utilisant des données audio étiquetées limitées et bégayantes.

Advancements in spoken language technologies for neurodegenerative speech disorders are crucial for meeting both clinical and technological needs. This overview paper is vital for advancing the field, as it presents a comprehensive review of state-of-the-art methods in pathological speech detection, automatic speech recognition, pathological speech intelligibility enhancement, intelligibility and severity assessment, and data augmentation approaches for pathological speech. It also highlights key challenges, such as ensuring robustness, privacy, and interpretability. The paper concludes by exploring promising future directions, including the adoption of multimodal approaches and the integration of large language models to further advance speech technologies for neurodegenerative speech disorders.

The adoption of advanced deep learning architectures in stuttering detection (SD) tasks is challenging due to the limited size of the available datasets. To this end, this work introduces the application of speech embeddings extracted from pre-trained deep learning models trained on large audio datasets for different tasks. In particular, we explore audio representations obtained using emphasized channel attention, propagation, and aggregation time delay neural network (ECAPA-TDNN) and Wav2Vec2.0 models trained on VoxCeleb and LibriSpeech datasets respectively. After extracting the embeddings, we benchmark with several traditional classifiers, such as the K-nearest neighbour (KNN), Gaussian naive Bayes, and neural network, for the SD tasks. In comparison to the standard SD systems trained only on the limited SEP-28k dataset, we obtain a relative improvement of 12.08%, 28.71%, 37.9% in terms of unweighted average recall (UAR) over the baselines. Finally, we have shown that combining two embeddings and concatenating multiple layers of Wav2Vec2.0 can further improve the UAR by up to 2.60% and 6.32% respectively.

Stuttering is a neuro-developmental speech impairment characterized by uncontrolled utterances (interjections) and core behaviors (blocks, repetitions, and prolongations), and is caused by the failure of speech sensorimotors. Due to its complex nature, stuttering detection (SD) is a difficult task. If detected at an early stage, it could facilitate speech therapists to observe and rectify the speech patterns of persons who stutter (PWS). The stuttered speech of PWS is usually available in limited amounts and is highly imbalanced. To this end, we address the class imbalance problem in the SD domain via a multibranching (MB) scheme and by weighting the contribution of classes in the overall loss function, resulting in a huge improvement in stuttering classes on the SEP-28k dataset over the baseline (StutterNet). To tackle data scarcity, we investigate the effectiveness of data augmentation on top of a multi-branched training scheme. The augmented training outperforms the MB StutterNet (clean) by a relative margin of 4.18% in macro F1-score $(\mathcal{F}_1)$ . In addition, we propose a multi-contextual (MC) StutterNet, which exploits different contexts of the stuttered speech, resulting in an overall improvement of 4.48% in $(\mathcal{F}_1)$ over the single context based MB StutterNet. Finally, we have shown that applying data augmentation in the cross-corpora scenario can improve the overall SD performance by a relative margin of 13.23% in F1 over the clean training.

Stuttering is a speech disorder during which the flow of speech is interrupted by involuntary pauses and repetition of sounds. Stuttering identification is an interesting interdisciplinary domain research problem which involves pathology, psychology, acoustics, and signal processing that makes it hard and complicated to detect. Recent developments in machine and deep learning have dramatically revolutionized speech domain, however minimal attention has been given to stuttering identification. This work fills the gap by trying to bring researchers together from interdisciplinary fields. In this paper, we review comprehensively acoustic features, statistical and deep learning based stuttering/disfluency classification methods. We also present several challenges and possible future directions.

The adoption of advanced deep learning (DL) architecture in stuttering detection (SD) tasks is challenging due to the limited size of the available datasets. To this end, this work introduces the application of speech embeddings extracted with pre-trained deep models trained on massive audio datasets for different tasks. In particular, we explore audio representations obtained using emphasized channel attention, propagation, and aggregation-time-delay neural network (ECAPA-TDNN) and Wav2Vec2.0 model trained on VoxCeleb and LibriSpeech datasets respectively. After extracting the embeddings, we benchmark with several traditional classifiers, such as a knearest neighbor, Gaussian naive Bayes, and neural network, for the stuttering detection tasks. In comparison to the standard SD system trained only on the limited SEP-28k dataset, we obtain a relative improvement of 16.74% in terms of overall accuracy over baseline. Finally, we have shown that combining two embeddings and concatenating multiple layers of Wav2Vec2.0 can further improve SD performance up to 1% and 2.64% respectively.