SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation
Authors:
Preprint
Abstract
We present a novel Speech Augmented Language Model (SALM) with {\em multitask} and {\em in-context} learning capabilities. SALM comprises a frozen text LLM, a audio encoder, a modality adapter module, and LoRA layers to accommodate speech input and associated task instructions. The unified SALM not only achieves performance on par with task-specific Conformer baselines for Automatic Speech Recognition (ASR) and Speech Translation (AST), but also exhibits zero-shot in-context learning capabilities, demonstrated through keyword-boosting task for ASR and AST. Moreover, {\em speech supervised in-context training} is proposed to bridge the gap between LLM training and downstream speech tasks, which further boosts the in-context learning ability of speech-to-text models. Proposed model is open-sourced via NeMo toolkit.Keywords
All Related Versions
This publication has 17 references indexed in Scilit:
- Neural Codec Language Models are Zero-Shot Text to Speech SynthesizersIEEE Transactions on Audio, Speech and Language Processing, 2025
- Prompting Large Language Models with Speech Recognition Abilities2024
- Adapting Large Language Model with Speech for Fully Formatted End-to-End Speech Recognition2024
- AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking HeadProceedings of the AAAI Conference on Artificial Intelligence, 2024
- SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual DataIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024
- Fast Conformer With Linearly Scalable Attention For Efficient Speech Recognition2023
- On Decoder-Only Architecture For Speech-to-Text and Large Language Model Integration2023
- Large-Scale Language Model Rescoring on Long-Form Data2023
- Integrating Text Inputs for Training and Adapting RNN Transducer ASR Models2022
- Accelerating RNN Transducer Inference via Adaptive Expansion SearchIEEE Signal Processing Letters, 2020