Yaqi Chen

Recent advancements in end-to-end (E2E) modeling have notably improved automatic speech recognition (ASR) systems; however, far-field speech recognition (FSR) remains challenging due to signal degradation from factors such as low signal-to-noise ratio, reverberation, and interfering sounds. This requires richer training data and multi-channel speech enhancement. To address this gap, we introduce Mixer, a novel data augmentation technique designed to further enhance the performance of large-scale pre-trained models for FSR. Mixer interpolates and mixes feature representations of speech samples and their corresponding losses, extending the MixSpeech framework to intermediate layers of Whisper. Additionally, we propose Mixer-C, which further leverages multi-channel information by combining speech from different microphone channels using a channel selector. Experimental results demonstrate that Mixer significantly outperforms existing methods, including SpecAugment, achieving a relative word error rate (WER) reduction of 3.6% compared to the baseline. Furthermore, Mixer-C offers an additional WER improvement of 2.2%, showcasing its efficacy in improving FSR accuracy.

Recent advancements in large-scale pre-trained automatic speech recognition (ASR) foundation models (e.g., Whisper) have exhibited remarkable performance in speech processing tasks. A recently emerging paradigm, prompt tuning, offers a parameter-efficient approach for fine-tuning, which has proven to be effective in enhancing the adaptation of pre-trained models to downstream tasks. In this paper, we first explore the prompting method for low-resource speech recognition based on Whisper. Although effective, it poses a challenge in the few-shot scenario due to its high sensitivity to initialization. To address this problem, we propose a novel meta-prompt for low-resource speech recognition that leverages the benefits of meta-learning for fast learning. Moreover, we further present a lightweight version of meta-prompt that omits the learning of encoder-prompt, reducing computational and storage costs. Extensive experiments on FLEURS datasets demonstrate consistent improvements across eleven target languages, showing better generalizability. Notably, meta-prompt achieves similar performance with a 20 $\%$ -shot compared to prompt tuning with a 50 $\%$ -shot setting, suggesting excellent few-shot learning ability.

Recent advancements in End-to-end (E2E) modeling have notably improved automatic speech recognition (ASR) systems; however, far-field speech recognition (FSR) remains challenging due to signal degradation from factors such as low signal-to-noise ratio, reverberation, and interfering sounds. This requires richer training data and multi-channel speech enhancement. To address this gap, we introduce Mixer, a novel data augmentation technique designed to further enhance the performance of large-scale pre-trained models for FSR. Mixer interpolates and mixes feature representations of speech samples and their corresponding losses, extending the MixSpeech framework to intermediate layers of Whisper. Additionally, we propose Mixer-C, which further leverages multi-channel information by combining speech from different microphone channels using a channel selector. Experimental results demonstrate that Mixer significantly outperforms existing methods, including SpecAugment, achieving a relative word error rate (WER) reduction of 3.6% compared to the baseline. Furthermore, Mixer-C offers an additional WER improvement of 2.2%, showcasing its efficacy in improving FSR accuracy.

Low-resource automatic speech recognition is a challenging task due to a lack of labeled training data. To resolve this issue, multilingual meta-learning learns a better model initialization from many source language tasks for fast adaptation to unseen target languages. However, for diverse source languages, the quantity and difficulty vary greatly because of their different data scales and phonological systems. These differences lead to task-quantity and task-difficulty imbalance issues and thus a failure of multilingual meta-learning ASR. In this work, we propose a task-based meta focal loss (TMFL) approach to address this tough challenge. Specifically, we introduce a hard-task moderator and update the meta-parameters using gradients from both the support set and query set. Our proposed approach focuses more on hard tasks and makes full use of the data from hard tasks. Moreover, we analyze the significance of the hard task moderator and interpret its significance at the sample level. Experiment results show that the proposed method, TMFL, significantly outperforms the state-of-the-art multilingual meta-learning on all target languages for the IARPA BABEL and OpenSLR datasets, especially under very-low-resource conditions. In particular, it can reduce character error rate from 72% to 60% by fine-tuning the pre-trained model with about 22 hours of Vietnamese data.