Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
Authors:
Top Cited Papers
- 1 May 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 6419-6423
Abstract
We present Mockingjay as a new speech representation learning approach, where bidirectional Transformer encoders are pre-trained on a large amount of unlabeled speech. Previous speech representation methods learn through conditioning on past frames and predicting information about future frames. Whereas Mockingjay is designed to predict the current frame through jointly conditioning on both past and future contexts. The Mockingjay representation improves performance for a wide range of downstream tasks, including phoneme classification, speaker recognition, and sentiment classification on spoken content, while outperforming other approaches. Mockingjay is empirically powerful and can be fine-tuned with downstream models, with only 2 epochs we further improve performance dramatically. In a low resource setting with only 0.1% of labeled data, we outperform the result of Mel-features that uses all 100% labeled data.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- wav2vec: Unsupervised Pre-Training for Speech Recognition2019
- Unsupervised Speech Representation Learning Using WaveNet AutoencodersIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2019
- Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech2018
- Self-Attentional Acoustic Models2018
- Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph2018
- Deep Contextualized Word Representations2018
- Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi2017
- Audio Word2Vec: Unsupervised Learning of Audio Segment Representations Using Sequence-to-Sequence Autoencoder2016
- Librispeech: An ASR corpus based on public domain audio books2015
- “Cloze Procedure”: A New Tool for Measuring ReadabilityJournalism Quarterly, 1953