Wav2Seq: Pre-Training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages
Authors:
- 4 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
We introduce Wav2Seq, the first self-supervised approach to pre-train both parts of encoder-decoder models for speech data. We induce a pseudo language as a compact discrete representation, and formulate a self-supervised pseudo speech recognition task — transcribing audio inputs into pseudo subword sequences. This process stands on its own, or can be applied as low-cost second-stage pre-training. We experiment with automatic speech recognition (ASR), spoken named entity recognition, and speech-to-text translation. We set new state-of-the-art results for end-to-end spoken named entity recognition, and show consistent improvements on 8 language pairs for speech-to-text translation, even when competing methods use additional text data for training. On ASR, our approach enables encoder-decoder methods to benefit from pre-training for all parts of the network, and shows comparable performance to highly optimized recent methods.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Libri-Light: A Benchmark for ASR with Limited or No Supervision2020
- wav2vec: Unsupervised Pre-Training for Speech Recognition2019
- Wav2Letter++: A Fast Open-source Speech Recognition System2019
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling2019
- End-To-End Named Entity And Semantic Concept Extraction From Speech2018
- ESPnet: End-to-End Speech Processing Toolkit2018
- Using the Output Embedding to Improve Language Models2017
- Listen, attend and spell: A neural network for large vocabulary conversational speech recognition2016
- Librispeech: An ASR corpus based on public domain audio books2015
- Web-scale k-means clustering2010