Self-Supervised Training of Speaker Encoder With Multi-Modal Diverse Positive Pairs
Open Access
- 20 April 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Vol. 31 (23299290) , 1706-1719
Abstract
We study a novel neural speaker encoder and its training strategies for speaker recognition without using any identity labels. The speaker encoder is trained to extract a fixed dimensional speaker embedding from a spoken utterance of variable length. Contrastive learning is a typical self-supervised learning technique. However, the contrastive learning of the speaker encoder depends very much on the sampling strategy of positive and negative pairs. It is common that we sample a positive pair of segments from the same utterance. Unfortunately, such a strategy, denoted as poor-man's positive pairs (PPP), lacks the necessary diversity. In this work, we propose a multi-modal contrastive learning technique with novel sampling strategies. By cross-referencing between speech and face data, we find diverse positive pairs (DPP) for contrastive learning, thus improving the robustness of speaker encoder. We train the speaker encoder on the VoxCeleb2 dataset without any speaker labels, and achieve an equal error rate (EER) of 2.89%, 3.17% and 6.27% under the proposed progressive clustering strategy, and an EER of 1.44%, 1.77% and 3.27% under the two-stage learning strategy with pseudo labels, on the three test sets of VoxCeleb1. This novel solution outperforms the state-of-the-art self-supervised learning methods by a large margin, at the same time, achieves comparable results with the supervised learning counterpart. We also evaluate our self-supervised learning technique on the LRS2 and LRW datasets, where speaker information is unavailable. All experiments suggest that the proposed neural architecture and sampling strategies are robust across datasets.Keywords
All Related Versions
Funding Information
- Huawei Noah's Ark Lab
- National Natural Science Foundation of China (62271432)
- Internal Project Fund from Shenzhen Research Institute of Big Data (T00120220002)
- Human-Robot Collaborative AI for Advanced Manufacturing and Engineering (A18A2b0046)
- Science, Technology and Research, Singapore
This publication has 49 references indexed in Scilit:
- VoxCeleb: A Large-Scale Speaker Identification Dataset2017
- Analysis of Score Normalization in Multilingual Speaker Recognition2017
- Lip Reading in the Wild2017
- A study on data augmentation of reverberant speech for robust speech recognition2017
- The Speakers in the Wild (SITW) Speaker Recognition Database2016
- Deep Residual Learning for Image Recognition2016
- An overview of text-independent speaker recognition: From features to supervectorsSpeech Communication, 2010
- Curriculum learning2009
- Combining labeled and unlabeled data with co-training1998
- Least squares quantization in PCMIEEE Transactions on Information Theory, 1982