Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?
Authors:
Preprint
Abstract
Self-supervised learning (SSL) has recently allowed leveraging large datasets of unlabeled speech signals to reach impressive performance on speech tasks using only small amounts of annotated data. The high number of proposed approaches fostered the need and rise of extended benchmarks that evaluate their performance on a set of downstream tasks exploring various aspects of the speech signal. However, and while the number of considered tasks has been growing, most rely upon a single decoding architecture that maps the frozen SSL representations to the downstream labels. This work investigates the robustness of such benchmarking results to changes in the decoder architecture. Interestingly, it appears that varying the architecture of the downstream decoder leads to significant variations in the leaderboards of most tasks. Concerningly, our study reveals that benchmarking using limited decoders may cause a counterproductive increase in the sizes of the developed SSL models.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Self-Supervised Speech Representation Learning: A ReviewIEEE Journal of Selected Topics in Signal Processing, 2022
- Pretext Tasks Selection for Multitask Self-Supervised Audio Representation LearningIEEE Journal of Selected Topics in Signal Processing, 2022
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingIEEE Journal of Selected Topics in Signal Processing, 2022
- Distilhubert: Speech Representation Learning by Layer-Wise Distillation of Hidden-Unit Bert2022
- Contrastive Learning of General-Purpose Audio Representations2021
- Libri-Light: A Benchmark for ASR with Limited or No Supervision2020
- Additive Margin Softmax for Face VerificationIEEE Signal Processing Letters, 2018
- X-Vectors: Robust DNN Embeddings for Speaker Recognition2018
- Librispeech: An ASR corpus based on public domain audio books2015
- The Buckeye corpus of conversational speech: labeling conventions and a test of transcriber reliabilitySpeech Communication, 2005