SALM: Speech-Augmented Language Model with in-Context Learning for Speech Recognition and Translation
Authors:
Top Cited Papers
- 14 April 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 13521-13525
Abstract
We present a novel Speech Augmented Language Model (SALM) with multitask and in-context learning capabilities. SALM comprises a frozen text LLM, a audio encoder, a modality adapter module, and LoRA layers to accommodate speech input and associated task instructions. The unified SALM not only achieves performance on par with task-specific Conformer baselines for Automatic Speech Recognition (ASR) and Speech Translation (AST), but also exhibits zero-shot in-context learning capabilities, demonstrated through keyword-boosting task for ASR and AST. Moreover, speech supervised in-context training is proposed to bridge the gap between LLM training and downstream speech tasks, which further boosts the in-context learning ability of speech-to-text models. Proposed model is open-sourced via NeMo toolkit 1 .Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Adapting Large Language Model with Speech for Fully Formatted End-to-End Speech Recognition2024
- Generative Speech Recognition Error Correction With Large Language Models and Task-Activating Prompting2023
- Fast Conformer With Linearly Scalable Attention For Efficient Speech Recognition2023
- On Decoder-Only Architecture For Speech-to-Text and Large Language Model Integration2023
- Large-Scale Language Model Rescoring on Long-Form Data2023
- Integrating Text Inputs for Training and Adapting RNN Transducer ASR Models2022
- MuST-C: A multilingual corpus for end-to-end speech translationComputer Speech & Language, 2021
- Accelerating RNN Transducer Inference via Adaptive Expansion SearchIEEE Signal Processing Letters, 2020
- Deep Context: End-to-end Contextual Speech Recognition2018
- Librispeech: An ASR corpus based on public domain audio books2015