Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining
Authors:
- 6 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7468-7472
Abstract
Much recent work on Spoken Language Understanding (SLU) is limited in at least one of three ways: models were trained on oracle text input and neglected ASR errors, models were trained to predict only intents without the slot values, or models were trained on a large amount of in-house data. In this paper, we propose a clean and general framework to learn semantics directly from speech with semi-supervision from transcribed or untranscribed speech to address these issues. Our framework is built upon pretrained end-to-end (E2E) ASR and self-supervised language models, such as BERT, and fine-tuned on a limited amount of target SLU data. We study two semi-supervised settings for the ASR component: supervised pretraining on transcribed speech, and unsupervised pretraining by replacing the ASR encoder with self-supervised speech representations, such as wav2vec. In parallel, we identify two essential criteria for evaluating SLU models: environmental noise-robustness and E2E semantics evaluation. Experiments on ATIS show that our SLU framework with speech as input can perform on par with those using oracle text as input in semantics understanding, even though environmental noise is present and a limited amount of labeled semantics data is available for training.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Is ATIS Too Shallow to Go Deeper for Benchmarking Spoken Language Understanding Models?2018
- Towards End-to-end Spoken Language Understanding2018
- Hybrid CTC/Attention Architecture for End-to-End Speech RecognitionIEEE Journal of Selected Topics in Signal Processing, 2017
- Encoder-decoder with focus-mechanism for sequence labelling based spoken language understanding2017
- Listen, attend and spell: A neural network for large vocabulary conversational speech recognition2016
- Librispeech: An ASR corpus based on public domain audio books2015
- End-to-end learning of semantic role labeling using recurrent neural networks2015
- What is left to be understood in ATIS?2010
- Multilingual spoken-language understanding in the MIT Voyager systemSpeech Communication, 1995
- The ATIS spoken language systems pilot corpus1990