Retrieval Augmented End-to-End Spoken Dialog Models
Authors:
- 14 April 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 12056-12060
Abstract
We recently developed a joint speech and language model (SLM [1]) which fuses a pretrained foundational speech model and a large language model (LLM), while preserving the in-context learning capability intrinsic to the pretrained LLM. In this paper, we apply SLM to dialog applications where the dialog states are inferred directly from the audio signal.Task-oriented dialogs often contain domain-specific entities, i.e., restaurants, hotels, train stations, and city names, which are difficult to recognize, however, critical for the downstream applications. Inspired by the RAG (retrieval-augmented generation) models, we propose a retrieval augmented SLM (ReSLM) that overcomes this weakness. We first train a retriever to retrieve text entities given audio inputs. The retrieved entities are then added as text inputs to the underlying LLM to bias model predictions. We evaluated ReSLM on speech MultiWoz task (DSTC-11 Challenge), and found that the retrieval augmentation boosts model performance, achieving joint goal accuracy (38.6% vs 32.7%), slot error rate (20.6% vs 24.8%) and ASR word error rate (5.5% vs 6.7%). While demonstrated on dialog state tracking, our approach is broadly applicable to speech tasks requiring custom contextual information or domain-specific entities.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- SLM: Bridge the Thin Gap Between Speech and Text Foundation Models2023
- AudioLM: A Language Modeling Approach to Audio GenerationIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023
- Wav2Seq: Pre-Training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages2023
- Accelerating RNN-T Training and Inference Using CTC Guidance2023
- Integrating Text Inputs for Training and Adapting RNN Transducer ASR Models2022
- Tts4pretrain 2.0: Advancing the use of Text and Speech in ASR Pretraining with Consistency and Contrastive Losses2022
- Controllable Semantic Parsing via Retrieval Augmentation2021
- Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue DatasetProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Dense Passage Retrieval for Open-Domain Question Answering2020
- Connectionist temporal classification2006