Tuning Large Language Model for Speech Recognition With Mixed-Scale Re-Tokenization
- 27 June 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 31 (10709908) , 1740-1744
Abstract
Large Language Models (LLMs) have proven successful across a spectrum of speech-related tasks, such as speech recognition, text-to-speech, and spoken language understanding. Recently, the use of discretized speech features has gained attention as an efficient and compatible alternative to continuous features for LLMs. This is mainly due to their reduced storage requirements and better alignment of these features with LLM's input space. However, the typical practice of freezing the speech encoder during training poses challenges in bridging the modality gap between speech and text. To address this, we propose to use a mixed-scale re-tokenization layer, integrating multiple granularities in discretized speech features directly within the LLM's input module. Our experimental results demonstrated that the proposed method can effectively enhance the performance of ASR in the setting of continuous learning of an LLM, highlighting the importance of a meticulously designed input module for the integration of discretized speech features with an LLM.Keywords
This publication has 22 references indexed in Scilit:
- Loss Masking Is Not Needed In Decoder-Only Transformer For Discrete-Token-Based ASR2024
- Generative Speech Recognition Error Correction With Large Language Models and Task-Activating Prompting2023
- Token-Level Serialized Output Training for Joint Streaming ASR and ST Leveraging Textual Alignments2023
- On Decoder-Only Architecture For Speech-to-Text and Large Language Model Integration2023
- SLM: Bridge the Thin Gap Between Speech and Text Foundation Models2023
- MSViT: Dynamic Mixed-scale Tokenization for Vision Transformers2023
- Vision Transformers with Mixed-Resolution Tokenization2023
- Are Discrete Units Necessary for Spoken Language Modeling?IEEE Journal of Selected Topics in Signal Processing, 2022
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden UnitsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Librispeech: An ASR corpus based on public domain audio books2015