Weihao Jiang

Lip reading aims at recognizing what a speaker is saying in a silent video only based on visual information. With the development of deep learning, lip reading has attracted much attention. However, different people have different facial features, which leads to failure when the model is tested on unseen speakers. Most existing methods deal with this problem with fine-tuning or speaker adaptation methods which require the source domain data or the labels of the target domain. However, in real-world applications, due to privacy or storage issues and the difficulty in annotating the data, using these methods is not realistic. To cope with this problem, in this paper, we focus on test-time speaker adaptation for lip reading where the source data and the labels of target data are unavailable during adaptation. We propose a robust entropy minimization strategy to amplify the optimization of high-confident samples and reduce the influence of the low-confident samples. Furthermore, we propose an adaptive pseudo-label strategy to select more confident samples between the outputs from the source model and the augmented-averaged teacher model. Extensive experiments show that our method outperforms all the other baselines consistently, demonstrating its effectiveness and universality.

In recent years, speech recognition has made significant progress, but recognizing accented speech remains a challenge. Although multi-accent speech recognition models have exhibited remarkable capabilities across different accents, their performance still degrades when encountering low-resource accents. In this paper, we propose AccentFusion, a framework leveraging accent similarity to improve cross-accent speech recognition. AccentFusion employs an interaction-augmented module to capture accent similarities between source and target accents with fine-grained association. Additionally, we use fusion-guided loss to supervise the weights of the target accent while learning accent similarity, encouraging the model to focus on its primary attention on the target accent. During the inference, we fuse the source and target accent features based on the similarity. We evaluate AccentFusion on the CommonVoice corpus. Experiments demonstrate that fusing accent information improves over fine-tuning baseline, significantly reducing word error rates (WER) on low-resource accents.