Adapting GPT, GPT-2 and BERT Language Models for Speech Recognition
Authors:
- 13 December 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 162-168
Abstract
Language models (LMs) pre-trained on massive amounts of text, in particular bidirectional encoder representations from Transformers (BERT), generative pre-training (GPT), and GPT-2, have become a key technology for many natural language processing tasks. In this paper, we present results using fine-tuned GPT, GPT-2, and their combination for automatic speech recognition (ASR). Unlike unidirectional LM GPT and GPT-2, BERT is bidirectional whose direct product of the output probabilities is no longer a valid language prior probability. A conversion method is proposed to compute the correct language prior probability based on bidirectional LM outputs in a mathematically exact way. Experimental results on the widely used AMI and Switchboard ASR tasks showed that the combination of the fine-tuned GPT and GPT-2 outperformed the combination of three neural LMs with different architectures trained from scratch on the in-domain text by up to a 12% relative word error rate reduction (WERR). Furthermore, on the AMI corpus, the proposed conversion for language prior probabilities enables BERT to obtain an extra 3% relative WERR, and the combination of BERT, GPT and GPT-2 results in further improvements.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Deep Contextualized Word Representations2018
- Approaches for Neural-Network Language Model Adaptation2017
- Purely Sequence-Trained Neural Networks for ASR Based on Lattice-Free MMI2016
- Neural Machine Translation of Rare Words with Subword Units2016
- Aligning Books and Movies: Towards Story-Like Visual Explanations by Watching Movies and Reading Books2015
- Bidirectional recurrent neural network language models for automatic speech recognition2015
- Extensions of recurrent neural network language model2011
- Continuous space language modelsComputer Speech & Language, 2007
- Completely Derandomized Self-Adaptation in Evolution StrategiesEvolutionary Computation, 2001
- Using out-of-domain data to improve in-domain language modelsIEEE Signal Processing Letters, 1997