Adapting Large Language Model with Speech for Fully Formatted End-to-End Speech Recognition
Authors:
- 14 April 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 11046-11050
Abstract
Most end-to-end (E2E) speech recognition models are composed of encoder and decoder blocks that perform acoustic and language modeling functions. Pretrained large language models (LLMs) have the potential to improve the performance of E2E ASR. However, integrating a pretrained language model into an E2E speech recognition model has shown limited benefits due to the mismatches between text-based LLMs and those used in E2E ASR. In this paper, we explore an alternative approach by adapting a pretrained LLMs to speech. Our experiments on fully-formatted E2E ASR transcription tasks across various domains demonstrate that our approach can effectively leverage the strengths of pretrained LLMs to produce more readable ASR transcriptions. Our model, which is based on the pretrained large language models with either an encoder-decoder or decoder-only structure, surpasses strong ASR models such as Whisper 1 , in terms of recognition error rate, considering formats like punctuation and capitalization as well.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Massively Multilingual Shallow Fusion with Large Language Models2023
- Accelerating RNN-T Training and Inference Using CTC Guidance2023
- Improving Readability for Automatic Speech Recognition TranscriptionACM Transactions on Asian and Low-Resource Language Information Processing, 2023
- Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization2023
- Unified Speech-Text Pre-training for Speech Translation and Recognition2022
- A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks2021
- Multilingual Speech Translation from Efficient Finetuning of Pretrained Models2021
- A Neural Grammatical Error Correction System Built On Better Pre-training and Sequential Transfer Learning2019
- A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition2018
- Towards Better Decoding and Language Model Integration in Sequence to Sequence Models2017