Speaker Adaptation for Lip Reading with Robust Entropy Minimization and Adaptive Pseudo Labels
Open Access
- 12 April 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-6
Abstract
Lip reading aims at recognizing what a speaker is saying in a silent video only based on visual information. With the development of deep learning, lip reading has attracted much attention. However, different people have different facial features, which leads to failure when the model is tested on unseen speakers. Most existing methods deal with this problem with fine-tuning or speaker adaptation methods which require the source domain data or the labels of the target domain. However, in real-world applications, due to privacy or storage issues and the difficulty in annotating the data, using these methods is not realistic. To cope with this problem, in this paper, we focus on test-time speaker adaptation for lip reading where the source data and the labels of target data are unavailable during adaptation. We propose a robust entropy minimization strategy to amplify the optimization of high-confident samples and reduce the influence of the low-confident samples. Furthermore, we propose an adaptive pseudo-label strategy to select more confident samples between the outputs from the source model and the augmented-averaged teacher model. Extensive experiments show that our method outperforms all the other baselines consistently, demonstrating its effectiveness and universality.Keywords
This publication has 27 references indexed in Scilit:
- Unsupervised Domain Adaptation for Speech Recognition via Uncertainty Driven Self-Training2021
- Learning from the Master: Distilling Cross-modal Advanced Knowledge for Lip Reading2021
- Speaker Adaptive Training Using Model Agnostic Meta-Learning2019
- Adversarial Speaker Adaptation2019
- X-Vectors: Robust DNN Embeddings for Speaker Recognition2018
- Deep neural networks for small footprint text-dependent speaker verification2014
- Speaker adaptation of neural network acoustic models using i-vectors2013
- KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition2013
- Speaker adaptation of context dependent deep neural networks2013
- Feature engineering in Context-Dependent Deep Neural Networks for conversational speech transcription2011