Self-supervised learning of speech representations with Dutch archival data
Preprint
Abstract
This paper explores the use of Dutch archival television broadcast data for self-supervised learning of speech foundation models, specifically wav2vec 2.0. We first study data quality assumptions for pre-training, and show how music, noise and speaker overlap affect SSL convergence and downstream fine-tuning performance. Secondly, we explore effectively pre-processing strategies to convert the noisy broadcast dataset into a qualitative dataset for pre-training, by using Whisper and WhisperX., Thirdly, we compare mono-lingual and multi-lingual pre-training with equivalent amounts of data, and show that mono-lingual pre-training is more robust to out-of-domain data. Lastly, we achieve a state-of-the-art LARGE wav2vec 2.0 model for the Dutch language, by a continuation of pre-training a wav2vec 2.0 XLS-R model checkpoint with our 55k hour archival dataset.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- A Comparative Analysis of Bilingual and Trilingual Wav2Vec Models for Automatic Speech Recognition in Multilingual Oral History Archives2024
- Combining Multilingual Resources and Models to Develop State-of-the-Art E2E ASR for Swedish2023
- Comparing Self-Supervised Pre-Training and Semi-Supervised Training for Speech Recognition in Languages with Weak Language Models2023
- Benefits of pre-trained mono- and cross-lingual speech representations for spoken language understanding of Dutch dysarthric speechEURASIP Journal on Audio, Speech, and Music Processing, 2023
- XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale2022
- WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech ProcessingIEEE Journal of Selected Topics in Signal Processing, 2022
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- MLS: A Large-Scale Multilingual Dataset for Speech Research2020
- JNAS: Japanese speech corpus for large vocabulary continuous speech recognition research.Acoustical Science and Technology, 1999